BLUF — for the decoder-only Transformers generation path tested here, left-pad unequal prompts, supply the real attention mask and extract new tokens after the common padded input width. An attention mask does not turn the final padded column into the final real token. Validate token-level behaviour before treating batching as a performance improvement.
A small pinned GPT-2 fixture on CPU reproduced the boundary: correct left padding preserved all four eight-token singleton continuations. Right padding changed one continuation; replacing the left-padding mask with all ones also changed one. Incorrect suffix slicing contaminated three results with input positions. These are local control-flow observations, not a production-model benchmark or a throughput claim.
Freeze the contract, not just the random seed
The buyer decision is whether a batching change preserves the existing generation contract. Keep model weights, tokenizer revision, prompt formatting, special tokens, output budget, decoding strategy, dtype, attention implementation and cache configuration fixed. Preserve request IDs through any sorting and restore the caller's order. Do not change truncation or a chat template while comparing batching policies.
The Hugging Face text-generation guide explicitly recommends left padding for decoder-only generation. Its padding reference distinguishes adding padding from removing input through truncation. A faster run that silently drops the end of a document is not evidence of a better batch policy.
This differs from embedding batch budgets: an encoder commonly pools attended hidden states, whereas a causal decoder must choose the next token from a particular sequence position. Copying an encoder's right-padding convention into a generation wrapper can change that decision. Serving engines with packed sequences or their own position-selection logic require separate tests; this article does not claim that every backend has the same dense-tensor contract.
Why a correct attention mask is not enough
In the installed Transformers 4.57.1 greedy/sampling implementation, the next-token path reads outputs.logits[:, -1, :]. The fixture checked that expression in the installed GenerationMixin._sample source. With unequal right-padded prompts, the last column of a shorter row is a padding position. Masking padding as attention keys does not instruct this generation loop to gather each row's last non-padding query position.
Left padding places the final real token at the last tensor column in every row. For the tested GPT-2 path, prepare_inputs_for_generation derives position IDs from the attention mask. That gives actual tokens appropriate logical positions despite their shifted tensor columns. Passing arbitrary position IDs, omitting the mask or reusing a cache built for another layout can break that relationship. Inspect your model adapter rather than prescribing one universal position-ID formula.
| Request | Padded input; width 5 | New suffix |
|---|---|---|
| Short | P P P A B | G … |
| Long | C D E F H | G … |
The layout is illustrative, not a tokenizer dump. P denotes padding and letters denote real input tokens. Both rows occupy width five; G is the first new token and is not part of the input width. The extraction boundary is column five for both requests, not the number of attended tokens in each row. Right padding would move P to the right of a short prompt while leaving the generation loop's last-column selection unchanged.
The executed fixture and its limits
Execution used sshleifer/tiny-gpt2 at this immutable revision, with model and tokenizer pinned to the same commit. It is a tiny two-dimensional GPT-2 test model, not an industrial assistant. Repetitive continuations are expected and cannot establish answer quality. The value of this fixture is exposing wrapper mechanics with real model execution.
The manifest records Python 3.12.13, PyTorch 2.15.0+cpu, Transformers 4.57.1, CPU float32, one PyTorch thread, eager attention and evaluation mode. There was no chat template; inputs were plain text. Generation used do_sample=False, num_beams=1, max_new_tokens=8 and use_cache=True. Pad and EOS IDs were both 50256. Explicit attention masks distinguished padding positions from attended content.
The four prompts were “Hello”, “The industrial pump requires inspection because the bearing temperature is”, “1, 2, 3,” and “A reliable system must”. Their token lengths were 1, 10, 6 and 4. Each ran alone, then all ran together at padded width ten. The same batch ran with right padding, and again with left padding but an intentionally incorrect all-ones mask. No customer text was used.
| Variant | Token parity with singleton | Interpretation |
|---|---|---|
| Left; real mask | 4 of 4 | Correct baseline in this fixture |
| Right; real mask | 3 of 4 | One changed continuation |
| Left; all-ones mask | 3 of 4 | One changed continuation |
| Slice at attended count | 1 of 4 clean suffixes | Three retain input positions |
For the fourth prompt, the singleton and correct left-padded run generated token ID 5087 eight times. Both mutations generated 16046 eight times instead. The other three prompts retained their singleton token sequences despite the mutations. That is a warning about test sensitivity: a passing example does not establish that a padding policy is correct.
Correct left padding still produced a maximum first-step absolute logit difference of approximately 7.45e-9 for the first prompt; its generated token IDs were unchanged. The other three first-step differences were zero in this run. For the fourth prompt, the right-padding difference was approximately 0.250236, and the all-ones-mask difference approximately 0.249780. These comparisons cover the first forward step, not every subsequent logit tensor.
Use exact comparison for emitted token IDs, and a separately declared numerical tolerance for tensor diagnostics. A starting diagnostic tolerance of 1e-6 absolute and 1e-5 relative may be reasonable for this float32 fixture, but it is a proposed test setting, not a universal release threshold. Near-tied logits can change an argmax even under small numerical differences. Greedy decoding removes random sampling; it does not promise bitwise invariance across hardware, precision, kernels or batch shapes.
Extract the suffix at the rectangular boundary
For this decoder-only generate API, returned sequences include the input tensor before the generated suffix. The implementation core is: inputs = tokenizer(prompts, padding=True, return_tensors="pt"); width = inputs["input_ids"].shape[1]; outputs = model.generate(**inputs, do_sample=False, num_beams=1, max_new_tokens=8); suffix = outputs[:, width:]. Configure tokenizer.padding_side="left" and an appropriate pad token first, and pass explicit pad/EOS settings for reproducibility.
Do not substitute attention_mask.sum() for width. In the tested width-ten batch, slicing each returned row at its attended length started too early for prompts one, three and four. It retained padding and/or prompt tokens. This is an output-assembly bug even when the model generated the right continuation. Compare token arrays before decoding: skip_special_tokens=True can conceal padding contamination without removing leaked real prompt tokens.
For mixed EOS completion, account for generated EOS and any subsequent output padding according to your response contract. Reusing EOS as the input pad ID does not by itself mean that left-padded input immediately stops generation. Nevertheless, masks and stop handling must be explicit. This fixture generated eight new tokens in every case; it did not exercise early EOS, streaming, beams or cancellation. Those remain release tests, not claimed passes.
Bound empty and near-limit inputs
The tested admission rule rejected a zero-token prompt rather than inventing a BOS policy. It reserved eight output positions within the model's 1,024-position limit. Repeating “ hello” created a 1,016-token accepted input; it executed and produced eight new tokens. The 1,017-token variant was rejected by the admission predicate before model execution. This verifies a local guard, not an HTTP rejection path.
A production chat service must count the fully formatted prompt, including template and special tokens. Reserve capacity using the effective model and serving limits, and account for padding-to-multiple policies where relevant. Do not silently truncate evidence to force acceptance. Document whether an empty user message is valid inside an otherwise nonempty chat template; that is different from a zero-token model input.
Decision table: locate the failing boundary
| Observed failure | Inspect first | Next action |
|---|---|---|
| Only shorter rows change | Padding side and final-column selection | Left-pad and repeat baseline |
| Left-padded outputs change | Mask, positions, dtype and cache | Isolate one mutation at a time |
| Prompt leaks into response | Suffix boundary | Slice at common input width |
| Same tokens; tiny logit drift | Argmax margin and numerical tolerance | Do not call it bitwise equality |
| Near-limit input fails | Formatted length plus output reserve | Reject explicitly; do not truncate silently |
| Reordered results | Request identity after sorting | Restore by immutable request ID |
Only after correctness passes should you benchmark scheduling, memory and latency on the intended hardware. Chunked prefill under mixed load addresses the separate scheduling decision. A correct left-padded batch can still waste rectangular space or delay short requests while waiting for peers. Length grouping trades lower padding against formation delay and operational complexity.
Release checklist and practical next step
Gate 1 — retain a versioned manifest with immutable model/tokenizer revisions, prompt-format digest, exact dependencies, dtype, device, attention backend, mask/position policy, cache settings, EOS/pad IDs and complete generation options. Keep an unchanged singleton baseline and immutable request IDs.
Gate 2 — compare mixed lengths, reversed order, repeated prompts and multiple batch compositions. Include deliberate right-padding and mask mutations so the suite demonstrates sensitivity. Compare raw input IDs, masks, positions, emitted suffix IDs and stop reasons. Store only approved synthetic or sanitized inputs; token IDs can reveal the original text and are not anonymization.
Gate 3 — test empty input, maximum formatted context, overflow rejection, early EOS, cancellation, result reordering and cache reuse. Separate executed cases from missing coverage. Token parity alone does not validate useful answers: apply the application's structured-output and semantic contract as a separate gate.
Gate 4 — measure end-to-end latency and completed valid responses at an agreed offered load. No throughput or speedup was measured in this fixture. Keep the simpler policy if the added batching machinery has no repeatable operational benefit, and roll back if any accepted request receives another request's suffix.
For a generation-path review, bring the pinned serving manifest, a sanitized mixed-length corpus and your current output contract. An implementation assessment can turn them into a reproducible accept-or-rollback test. The deliverable is evidence that batching preserves the decision boundary—not an assumption that more parallel requests must be better.
The generation API reference documents output and decoding controls. Current documentation may describe a newer release than this intentionally pinned fixture; validate options against the deployed version before adopting them.
