Reranker Token Budgets: Find Evidence Lost After Retrieval

The relevant passage can reach your candidate list and still disappear before the reranker scores it. Diagnose that boundary before changing embeddings or expanding the index. Freeze retrieval, inspect the exact query–document tokens entering the cross-encoder, and compare evidence survival with a no-reranker baseline. Increase pair length or introduce passage windows only when the resulting ranking gain justifies their latency and operating cost.

This is a different failure from retrieving the wrong manual. A reranker may receive a source containing the answer while its tokenizer retains only the introduction. The resulting score evaluates the surviving text, not the evidence your application believes it supplied. An apparently successful retrieval trace can therefore conceal a downstream evidence loss.

Locate the boundary that actually loses evidence

The Sentence Transformers retrieve-and-rerank guide separates first-stage candidate retrieval from cross-encoder scoring. A cross-encoder jointly processes the query and a candidate. The CrossEncoder API documents a maximum input length and truncation of longer inputs. Neither guarantees that the decisive sentence survives your deployed preprocessing.

Record evidence at four boundaries: the frozen candidate text, the tokenized pair, the selected ranked passages, and the generator context. A failure at each boundary requires a different intervention. If the right source never becomes a candidate, use the hybrid retrieval diagnostic for exact identifiers. If the evidence survives tokenization but drops below the final cutoff, investigate ranking quality and selection policy. If it reaches the final prompt yet the answer is wrong, inspect generation separately.

Do not rebuild document segmentation as the first response. The chunking evaluation baseline concerns an upstream design choice. Here, keep those chunks and their candidate order fixed so that the reranker experiment has an identifiable cause.

Treat the pair limit as a shared budget

The usable document budget is the pair limit minus query tokens, special tokens and any separately added instructions or prefixes. Count with the deployed tokenizer, after the exact production formatting. Character counts, embedding-tokenizer counts and an advertised generator context window are not substitutes. Do not subtract a prefix twice if it is already included in the measured query tokens.

Hugging Face's padding and truncation reference distinguishes longest_first, only_first and only_second. In a query-first, document-second pair, only_second protects the query by truncating the document. It is not universally safer: a query that consumes the available capacity may make that policy impossible. Define an explicit rejection or reviewed query-shortening path instead of silently dropping the part number or negation. The deployed wrapper may apply a different strategy; inspect its actual call and truncation side rather than assuming defaults.

Padding is separate from truncation. Padding adds positions for batching; truncation removes content. Preserve the attention mask, batch size, maximum padded length and unpadded lengths in diagnostic aggregates. Bucket similar-length pairs if measurements justify it, but verify that reordering for batching restores the original candidate IDs before results are joined.

An executed fixture: move the same evidence, not the task

For this article, a tokenizer-only synthetic fixture was executed using tokenizers 0.23.2 and the tokenizer from cross-encoder/ms-marco-MiniLM-L6-v2, pinned to revision 233902d25c440f23af6f7d6e94d2946bac0bee0a. No model weights were loaded. No relevance scores, nDCG results, latency benchmarks or production improvements were measured. This English-trained model's tokenizer is used to illustrate a boundary, not to recommend it for German ranking.

The invented evidence was “The approved seal for AX-104-B is ZQ-77.” It occupied 15 tokenizer tokens. Ninety repetitions of “Routine maintenance documentation contains general inspection information.” supplied controlled filler. The evidence was placed before all filler, after 45 repetitions, or after all 90. These identifiers are synthetic and convey no real maintenance instruction.

The short English query “Which seal fits AX-104-B?” used nine tokens. The deliberately longer German query “Welche freigegebene Dichtung passt zu AX-104-B unter Berücksichtigung der dokumentierten Revision und der vorgesehenen Betriebsbedingungen?” used 48. These are not matched translations: the difference does not measure a language penalty. The untruncated pairs were 747 and 786 tokens respectively, including special tokens.

These results were identical for longest_first and only_second in this fixture because the document remained the longer sequence. They do not establish equivalence for long queries or other models. The useful finding is narrow: increasing this limit rescued the middle evidence but not the tail. Increasing candidate count alone cannot restore a sentence removed from every scored pair.

Reproduce the token-survival inspection

Use a fast tokenizer with offsets and sequence identity. First encode the complete pair without truncation or padding. Mark the document evidence span by character offsets in the exact formatted document string. Collect every non-empty token offset belonging to sequence 1 that intersects that span. Then encode the same pair under the proposed limit and collect sequence-1 offsets again. Full survival means every target token offset remains. Count partial survival separately: retaining the identifier while losing the restriction is not a successful case.

The inspection script used here follows this reproducible core with the tokenizers Python API. Load the pinned tokenizer.json with Tokenizer.from_file; call no_truncation() and no_padding(); obtain full = tok.encode(query, doc). Build target from zip(full.offsets, full.sequence_ids), requiring sid == 1, a < evidence_end, b > evidence_start and b > a. Next call enable_truncation(max_length=limit, strategy=strategy, stride=0), encode again, and build kept from non-empty sequence-1 offsets. Report len(target & kept), len(target), and target <= kept. Compare only the primary encoding passed to the model, not overflow encodings that the application never scores.

Our full local script and results were retained with the publication evidence. Its tokenizer.json SHA-256 was d241a60d5e8f04cc1b2b3e9ef7a4921b27bf526d9f6050ab90f9267a1f9e5c66. Repeating the core with a different tokenizer is a new experiment. For a production release gate, also capture the actual wrapper's input_ids and attention_mask and compare them to the diagnostic encoding. A standalone tokenizer test does not prove what your serving wrapper sends.

Offsets require care. Prefixing a title changes evidence positions; normalization may change token boundaries; some pipelines lack usable offsets. Fail the inspection explicitly when mappings are unavailable rather than reporting zero loss. Store aggregate survival counts in routine telemetry. Retain restricted diagnostic fixtures under access control instead of logging customer documents or raw token IDs that can expose text.

Separate candidate count from pair length

Create a frozen manifest of query ID, corpus revision, candidate IDs and ordered texts, annotated evidence spans, tokenizer/model revisions, preprocessing version, truncation side, pair limit and final context-selection budget. Use three routes: unchanged retrieved order without reranking; the current cross-encoder policy; and the proposed longer-pair or window policy. Apply the same final context budget to all routes.

Run the pair-length sweep with the same candidate set first, then sweep candidate count separately. If you later run a joint grid, keep both axes explicit. More candidates mean more pairs; passage windows can multiply those pairs again. A larger limit may also change padded batch length. Measure the actual backend rather than extrapolating cost from token counts alone.

Report relevant evidence retained at the final cutoff with a clear denominator of annotated relevant evidence units. Report nDCG with a defined relevance scale and deterministic tie handling. Deduplicate by source evidence identity so that three windows containing one sentence do not become three successes. Include per-query latency distributions, queueing and warm/cold conditions, throughput, hardware, batching and an explicit padding-waste definition. Raw cross-encoder scores are not calibrated probabilities that the eventual answer is correct.

Windowing is a candidate policy, not a free repair

If a source is too long, create query-budget-aware passage windows without rebuilding the retrieval index. Keep parent document ID, revision, section, character span and permission scope. Overlap can preserve a warning split at a boundary, but it also increases computation and duplicate exposure. Determine the overlap from the evidence structures you must preserve, not an unexplained percentage.

Define how window scores become a candidate score. Maximum score is a reasonable experiment, not a guaranteed rule: documents with many windows get more opportunities for a high score, and a locally relevant sentence may omit an applicability condition elsewhere. Compare maximum scoring with a capped-window alternative and preserve adjacent constraints in the final passage selection. Never show a parent-document citation while discarding the revision or span that made the result verifiable.

Window extraction should use information available at inference. Oracle windows centered on annotated answers are useful diagnostic upper bounds, not deployable improvements. Keep them separately labelled so that an evaluation cannot accidentally promote an answer-aware selector.

A release checklist your team can own

Gate 1 — Replay fixed candidate fixtures with evidence at the beginning, middle and end; include query prefixes, long queries, exact identifiers, negations and evidence straddling a window boundary.

Gate 2 — Assert source/revision/permission identity survives window expansion and batching. Reject silent truncation-policy changes, missing offsets and unsupported model-length settings.

Gate 3 — Compare no-reranker, current and proposed routes at the same final context budget. Review evidence survival, ranking quality and tail latency by protected query segment, not only a pooled average.

Gate 4 — Promote only against thresholds agreed from your workload and risk budget. Retain a rollback to the previous reranker configuration. The RAG evaluation CI gate provides the broader release structure; the new fixture adds a token-boundary control rather than replacing end-to-end evaluation.

Questions before changing the serving configuration

Does a larger max_length always fix the problem? No. The chosen model and backend must support that length; the tail may still be cut, and additional distractors may worsen ranking. Validate survival and quality separately.

Should every query use only_second? No. It preserves the first sequence only when the requested pair is feasible. Query order, templates, query length and error handling are part of the contract.

Can a low score prove the source is irrelevant? No. Check which tokens were scored. Even a fully visible passage can receive a poor relevance score, and relevance is not answer correctness.

If your team is deciding between a new embedding model and a reranker change, start with a bounded diagnostic review: one frozen candidate snapshot, annotated evidence, a serving manifest and a latency budget. I can help design that evaluation and its release gates. The deliverable should be a reproducible decision about where evidence is lost—not another model migration justified by an opaque aggregate score.