A successful embedding response does not prove that the encoder received the complete question. Before changing retrieval models, verify that part numbers, negations and applicability conditions survive the exact query-tokenization path. Reject or clarify an input when a protected condition disappears. A longer context window, a shorter vector and a fluent query summary are three different mechanisms; none is evidence that the original constraint was preserved.
The executed synthetic fixture in this article makes that distinction concrete. Two questions that differed in their required part number produced identical model input IDs and identical embeddings when the distinguishing condition sat beyond the truncation boundary. Moving it to the beginning preserved the condition, but still did not make the first retrieved document applicable. Token survival is a necessary diagnostic, not a retrieval-quality guarantee.
Locate the loss before the first retrieval request
Consider a maintenance question with a long incident history followed by “Only ZX-41; do not use revision B.” These are invented identifiers, not maintenance instructions. If preprocessing removes that final clause, retrieval can return a plausible procedure for the wrong part or revision without any infrastructure error. The vector database never received the distinction that the user supplied.
This is upstream of the reranker token-budget boundary, where a query–document pair can lose evidence after initial retrieval. It is also different from document chunking evaluation: keeping better document fragments does not restore a missing query restriction. Hold those downstream components fixed while testing query admission.
A useful boundary diagram is: original request → formatted query and protected-span manifest → full tokenization → budget and span-survival check → encoder → retrieval → applicability validation. On a failed survival check, branch to rejection or structured clarification before calling the encoder. Keep the original request available under the application's retention policy; never silently replace it with the surviving prefix.
Freeze the real encoder contract
Record the model repository and revision, tokenizer files and library versions, wrapper version, exact prompt prefix, maximum sequence length, truncation side, pooling and normalization. Inspect the actual input_ids and attention_mask entering the model. A tokenizer test that bypasses the serving wrapper can validate the wrong string.
Hugging Face's truncation documentation distinguishes padding from truncation and exposes explicit strategies and maximum lengths. Padding adds positions; truncation removes content. In a single query, right truncation can discard the tail; left truncation can discard the beginning. Changing sides relocates the risk rather than solving it.
The SentenceTransformer API separates maximum input sequence length from truncate_dim, which reduces embedding dimensionality. Reducing the output vector dimension does not buy more input tokens. An advertised generator context window is not the embedding wrapper's budget either. Increasing max_seq_length beyond the supported model and backend configuration is not a validated repair.
Count the fully formatted input, including its actual prefix and special tokens. Do not estimate with characters or separately tokenize fragments and assume their counts always add: tokenizer boundaries can change when strings are joined. The tokenizer utility reference documents offset mapping for compatible fast tokenizers; unsupported implementations can raise an error. Verify the installed binding instead of assuming that every tokenizer exposes offsets.
Executed fixture: a retained-token audit, not a benchmark
The experiment used sentence-transformers/all-MiniLM-L6-v2 at pinned revision 1110a243fdf4 (full commit in the repository link below), on CPU in evaluation mode with two Torch threads. Runtime versions were Python 3.14.9, Sentence Transformers 5.1.2, Transformers 4.57.1, tokenizers 0.22.2 and Torch 2.15.0+cpu. The pinned model repository makes the model choice reproducible. This English encoder is a test vehicle, not a recommendation for German retrieval.
We explicitly configured a test-only query prompt of “query: ” to exercise prefix accounting. This is not a claim that this model requires that prefix. Its standalone encoding had two tokens; the model's sequence budget was 256 and its special-token overhead was two. The tokenizer used right truncation. The complete joined input was measured directly.
Each query repeated “maintenance procedure ” 266 times. The protected clause was either “Only ZX-41; do not use revision B.” or the otherwise identical ZX-42 clause. Each appeared once, either before or after the filler. All four formatted inputs contained 548 tokens; the model received 256. Full, untruncated tokenization was used only for inspection, never sent to the encoder.
| Clause position | ZX-41 query | ZX-42 query |
|---|---|---|
| Before filler | Complete clause retained | Complete clause retained |
| After filler | Entire clause lost | Entire clause lost |
| Late-pair comparison | Same retained input IDs | Same observed embedding |
| Applicable evidence at rank one | No, early and late | No, early and late |
A forward pre-hook on the underlying Transformer captured the input IDs used by encode_query. They exactly matched explicit tokenization of the prefixed query at max_length=256. The normalized embeddings also matched explicit forward execution and normalization within an absolute tolerance of 0.000001. Thus the test inspected the library wrapper's real execution path, not only a parallel tokenizer simulation. It did not exercise an HTTP gateway or a customer deployment.
For the late pair, every token overlapping the protected clause was absent. The two retained ID sequences were identical, and the observed output embeddings were identical. No downstream ranker operating only on those query vectors can reconstruct which part number was originally requested. Passing the original constraints separately to a later applicability filter is a different architecture and must be tested as such.
Preserved words still do not enforce a condition
The exact-search sanity check used four invented evidence sentences: ZX-41 revision A with B not applicable; the corresponding ZX-42 sentence; ZX-41 revision B; and ZX-42 revision B. The first two were the manually specified applicable evidence for their respective queries. All document vectors were scored directly against each query vector, with no approximate index.
The top result was the ZX-42 revision-B sentence in all four cases. None matched its required evidence. Even the two early queries, whose complete protected clauses survived, failed this tiny top-one check. Repeated filler and this encoder's similarity behavior are not a basis for a general model-quality estimate. The result is useful precisely because it prevents the false conclusion that “constraint retained” means “constraint obeyed.”
Use explicit metadata or validated structured conditions for exact part and revision requirements where the corpus supports them. Validate the applicable evidence after retrieval. A positive similarity score is not proof of negation handling, revision eligibility or safe execution. This fixture measured neither production recall nor latency, and no customer outcome is inferred from it.
Build the admission check around spans
First construct the exact formatted query and locate every protected clause in that string. In the fixture, the complete early clause occupied character span [7, 41); the late clause occupied [5860, 5894). Offsets refer to that prefixed string, not to the unformatted user text. Ambiguous or repeated substrings require explicit span identities rather than an unchecked first-match search.
Tokenize without truncation and collect each non-empty offset intersecting the protected span. Tokenize again under the deployed budget. Require every target offset to remain, and compare the retained token IDs and attention mask with the serving path. Check complete clauses, not just identifier tokens: retaining ZX-41 while dropping “do not use revision B” is a failure. Partial survival must not count as success.
If normalization, routing or a prompt template changes the string, regenerate the mapping at that boundary. With unavailable offsets or an opaque provider, do not report verified survival. A conservative alternative is to reject over-budget inputs until the path can be inspected, while separately testing provider limits and error behavior. Set a byte/character admission cap before full tokenization as a resource guard; this is not a substitute for the semantic token-budget check.
| Observed boundary | Admission decision | Required validation |
|---|---|---|
| Protected span lost or partial | Reject or ask for structured clarification | No embedding of silently altered intent |
| All spans retained; evidence wrong | Inspect applicability and ranking | Do not declare truncation repaired quality |
| Input rewritten or summarized | Revalidate before admission | Exact IDs plus reviewed negation and scope |
| Offsets unavailable | Mark survival unverified | Conservative length gate and path inspection |
| Larger supported budget proposed | Run controlled comparison | Survival, applicable evidence and cost |
Release checklist and operating cost
Gate 1 — replay protected clauses at the beginning, near the boundary and beyond it. Include partial negations, repeated identifiers, Unicode, naturally authored German questions and every supported prompt route. The executed fixture covered only two English clauses at two positions; these additional cases remain production acceptance work.
Gate 2 — assert wrapper parity for IDs, masks and formatting. Pin the full dependency lock and model revision. Fail explicitly when mappings disappear or the configured budget changes. Preserve restricted synthetic audit artifacts; do not put customer token IDs or raw queries into ordinary metrics because they can expose text.
Gate 3 — measure both admission behavior and applicable-evidence selection. Report accepted, rejected and clarified requests with denominators. Count false rejections against reviewed valid questions. Do not improve a quality percentage by silently removing difficult requests from the evaluation population.
Gate 4 — benchmark the added tokenization and validation work on the actual traffic shape. A second tokenizer pass has CPU and latency cost; caching needs the full model, tokenizer and formatting identity. Clarification increases user effort, while a larger supported encoder can increase inference cost and require index migration. Keep the previous policy available for rollback without reopening an unsafe truncation path.
Only after the semantic input contract is stable should you optimize embedding batch budgets. A faster batch is not an improvement if it embeds a different question. Conversely, rejecting every long request may protect constraints but make the product unusable; agree the clarification and structured-input flow with the service owner.
What to bring to a query-admission review
Bring the encoder manifest, a sanitized set of long questions, the protected conditions and a small reviewed evidence set. I can help your team turn those artifacts into a reproducible admission test, a clarification policy and a release gate. The useful deliverable is a defensible boundary between what the user asked and what the model actually received—not a promise that a larger context window will fix retrieval.
