A serving adapter can change the model input without changing a single message in your request JSON. If it renders a chat template and then adds special tokens again, the resulting token sequence may contain a duplicated beginning-of-sequence marker. Make exact token-ID parity a release gate before investigating model quality. In the tokenizer-only fixture below, the intended input contained 14 tokens; the extra special-token pass produced 15 and began with [1, 1] instead of [1]. This is measured serialization drift, not a measured accuracy loss.
The boundary that message-level tests miss
An enterprise gateway receives role/content messages, a model name and generation settings. The downstream model does not consume those dictionaries directly: it receives a token sequence. Template selection, whitespace, role delimiters and tokenizer post-processing determine that sequence. A gateway migration can therefore pass JSON-schema checks while violating the model's input contract.
Hugging Face's chat-template guide explicitly warns against adding special tokens after a template has already supplied them. Its recommended split path uses tokenize=False during formatting and add_special_tokens=False during subsequent encoding. Direct apply_chat_template with tokenize=True avoids that additional pass. This is not a universal ban on special tokens: it assigns responsibility for inserting them to one stage.
Treat the tokenizer, template and adapter as part of the LLM release bundle. A prompt-text checksum alone cannot prove that the serving boundary preserved the input. Store the template checksum and tokenizer revision beside the request-mode flags and compare the IDs produced by both code paths.
An executed, pinned tokenizer fixture
The fixture used mistralai/Mistral-7B-Instruct-v0.1 at the immutable revision linked below, Python 3.14.9, Transformers 4.57.1 and Tokenizers 0.22.2. Only tokenizer files were loaded; no model weights or generation engine ran. This older model is a compact, inspectable test case, not a recommendation for a new deployment. The pinned tokenizer configuration supplies the template used here.
The request was one user message: Return the word READY. Rendering produced <s> [INST] Return the word READY. [/INST]. The direct and correctly split paths both produced this exact ID list: [1, 733, 16289, 28793, 4571, 272, 1707, 24805, 28802, 28723, 733, 28748, 16289, 28793]. Positions below are zero-based. The BOS token was 1; EOS was 2. Do not copy these IDs to another tokenizer.
| Path / mutation | Token trace | Observed result |
|---|---|---|
| Direct template / correct split | 14 / 14; BOS at 0 | Exact ID parity |
| Extra special-token pass | 15; BOS at 0 and 1 | Duplicate BOS detected |
| Wrong appended [INST] | 16; changed suffix | Full-sequence mismatch |
| Generation prompt off | 14; unchanged IDs | Expected template-specific no-op |
| Prefill continued / closed | 16 without EOS / 17 with EOS | Different request modes |
| Two completed assistant turns | 29; EOS at 16 and 28 | Legitimate repeated EOS |
The wrong-prefix mutation appended [INST] after the rendered prompt, simulating an adapter that starts a user instruction where an assistant continuation is intended. It produced 16 tokens. Importantly, its earlier trailing tokenization also changed: string concatenation is not guaranteed to translate into appending a fixed number of token IDs. Compare entire sequences, not merely the final token or length.
The fixture also exercised an assistant prefill containing READY. continue_final_message=True yielded 16 tokens with no EOS; the normally closed assistant message yielded 17 with EOS at position 16. Combining continue_final_message=True and add_generation_prompt=True raised ValueError. A separate two-turn conversation legitimately contained EOS at positions 16 and 28. Rejecting any repeated special token would falsely reject this valid history.
Reproduce the parity check
Install the pinned libraries in an isolated environment: pip install transformers==4.57.1 tokenizers==0.22.2 jinja2==3.1.6. Load AutoTokenizer.from_pretrained with the model ID and exact revision above. Use messages = [{"role": "user", "content": "Return the word READY."}].
Compute direct = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True). Then rendered = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True), followed by split = tokenizer(rendered, add_special_tokens=False)["input_ids"]. Require assert direct == split. For the negative control, compute duplicated = tokenizer(rendered, add_special_tokens=True)["input_ids"] and require duplicated != direct plus duplicated[:2] == [tokenizer.bos_token_id, tokenizer.bos_token_id] for this specific fixture.
Do not generalize that last assertion to every model. Some tokenizers do not insert BOS on ordinary encoding; some templates use other boundaries. The tokenizer API reference documents these switches, but the tested library version and actual tokenizer files remain the executable contract. Current main-branch documentation can describe newer defaults than a pinned runtime.
For this template, toggling add_generation_prompt did not change the IDs. That is a passing model-specific expectation, not evidence that the flag never matters. A template that emits an assistant header needs a separate positive test that the intended header is present. If your adapter supports several model families, maintain a fixture per family rather than imposing one universal suffix.
A serving acceptance table
| Condition | Acceptance check | Release action |
|---|---|---|
| Two serialization paths | Exact IDs with fixed artifacts | Accept only matching contract |
| Duplicate boundary | Compare expected token positions | Block; fix owning stage |
| Repeated EOS across turns | Validate per-message structure | Allow valid repetition |
| Assistant header required | Model-specific expected suffix | Block wrong role prefix |
| Prefill continuation | Explicit mode and end boundary | Reject contradictory flags |
| Parity passes | Separate generation and task tests | Do not claim quality approval |
Use this table as a release checklist, not an automatic repair algorithm. Never deduplicate token IDs blindly: repeated boundaries may represent separate messages, and repeated ordinary tokens may be meaningful content. A mismatch should block the candidate adapter or route it to investigation, not silently rewrite the user's conversation.
Evidence to keep at the handoff
Checklist 1 — Freeze the artifact identity. Record model and tokenizer revisions, library versions, exact template bytes and SHA-256, tokenizer file hashes, adapter revision and any template override. Retain the computed digest in the test manifest alongside the exact template bytes. Pin by immutable revision rather than a moving branch name.
Checklist 2 — Declare the request mode. Record whether this is a new assistant response or continuation of an assistant prefill. Set add_generation_prompt and continue_final_message intentionally. Avoid inheriting a pipeline's role-dependent default when comparing it with a hand-written adapter.
Checklist 3 — Compare before batching. Preserve the unpadded token IDs, length, first differing position, special-token positions and attention mask where applicable. Then test padding and truncation independently. Matching an unpadded fixture does not validate a batched deployment or a prompt near its context limit.
Checklist 4 — Make the negative controls fail. Exercise the extra special-token pass, incorrect role prefix, closed-versus-continued prefill and incompatible flags. Include multi-turn histories to prove that legitimate repetition is accepted. Extend coverage for system messages, empty content and tool schemas only where the chosen template supports them.
Checklist 5 — Keep evidence proportionate. Synthetic fixtures can retain full token traces. Production token IDs can reconstruct sensitive text; they are not anonymized telemetry. Prefer restricted, short-lived diagnostics and aggregate mismatch counts over logging every customer prompt.
What passing does not prove
Token parity is a cheap deterministic gate relative to model generation, but this test did not measure latency, cost, output accuracy or production traffic. It did not run an inference framework because none was installed in the isolated fixture environment. Identical token IDs do not guarantee identical outputs across hardware, kernels or decoding settings.
Generation needs its own acceptance checks for EOS/stop handling, max_new_tokens, sampling and actual model revision; see the generation configuration reference. A request can have perfectly serialized input and still stop too early or exceed the intended output budget. Likewise, serving parity says nothing about which tokens contributed to training loss; that is the separate fine-tuning loss-mask check.
Exact fixtures also create maintenance work. An intentional template change should fail the old baseline. Review the token diff, run task-level evaluation and approve a new baseline rather than weakening the comparison. Connect this workflow to evaluation-driven prompt versioning, keeping input-contract tests separate from answer-quality metrics.
Put the gate at the adapter boundary
Before replacing a gateway or upgrading a tokenizer, select one real request shape per supported model family, pin the artifacts and compare the final serialized inputs. Add deliberate failures before declaring the gate useful. If your team needs a deployment review, bring the adapter code, tokenizer snapshot and a sanitized request fixture: these are enough to scope a concrete input-contract audit without collecting customer conversation histories.
