Embedding Batch Budgets: Measure Padding Waste Before Increasing Batch Size

BLUF — increase an embedding batch only after measuring how much of its rectangular input is padding, how long requests wait and whether outputs still belong to the correct request IDs. A full batch is not necessarily useful work. Length bucketing can remove padding, but it adds scheduling policy and does not guarantee a better tail latency.

For this article, a pinned MiniLM encoder was executed on CPU with the same synthetic arrival trace under three policies. The fixed-width batch filled every request slot while spending 74.32% of input positions on padding. Bucketing reduced that fraction to 1.87%, yet its p95 was worse than the singleton baseline in this run. Those are local measurements, not GPU economics or a production sizing recommendation.

Freeze the semantic contract before the batch policy

An encoder maps text to a vector; a batching change should not quietly change the text. Pin the model and tokenizer revisions, input formatting, maximum length, truncation side, dtype, attention mask, pooling and normalization. Keep input IDs associated with immutable request IDs throughout sorting, execution and result assembly. Equal output dimensions alone do not establish equivalence.

The Hugging Face padding reference separates padding, which adds positions, from truncation, which removes content. Reducing the maximum length to make a benchmark faster is not a batching optimization if evidence disappears. The reranker token-budget diagnostic addresses that evidence-loss boundary; here the decision is how to group unchanged encoder inputs.

Define useful input positions as the sum of attention-mask ones, including attended special tokens. Define allocated positions as the sum of batch size multiplied by padded sequence width over executions. Padding fraction is one minus useful positions divided by allocated positions. This is a tensor-occupancy metric, not a measurement of wasted FLOPs: kernel implementations, attention paths and hardware determine the real execution cost.

Keep request-slot occupancy separate. Eight requests in an eight-slot batch can be 100% full while most token positions are padding. Conversely, a partially filled bucket may do less total work and still finish later because it waited for compatible arrivals.

The executed CPU fixture

The model was Sentence Transformers' all-MiniLM-L6-v2 at this pinned revision, using its onnx/model.onnx and tokenizer.json. The model card describes 384-dimensional outputs, attention-mask-aware mean pooling and normalization, and a default truncation boundary of 256 word pieces. This English encoder is a reproducible fixture, not a recommendation for German retrieval quality.

Execution used float32, ONNX Runtime 1.31.0, tokenizers 0.23.3, NumPy 2.5.4 and Python 3.12.13. The Linux guest exposed affinity CPUs 0–3; the ORT session used one intra-op thread, one inter-op thread, sequential graph execution and disabled intra-op spinning. Four warm-up calls preceded the runs. The CPU thread-budget walkthrough explains why these controls must remain fixed rather than increasing threads alongside batch size.

The synthetic sentence “Inspect the bearing temperature.” was repeated 1, 32, 2, 16, 1, 32, 2 and 16 times per burst. Tokenized lengths, including special tokens, were 7, 162, 12, 82, 7, 162, 12 and 82. Eight identical bursts arrived 400 milliseconds apart, with requests within a burst one millisecond apart: 64 requests and 4,208 attended positions per policy. No measured input was truncated. A separate 302-token overflow fixture triggered the policy's over-256 rejection predicate; it was not sent to the model and is not an HTTP error-path test.

Singletons executed immediately when the worker became available, padded only to their own length. Fixed batching grouped up to eight requests and padded every batch to 256 positions. Bucketing grouped lengths into ceilings 32, 128 and 256, but padded each execution to its actual longest member, not the bucket ceiling. Both batching policies allowed up to 15 milliseconds of deliberate formation delay, with an earlier dispatch when eight eligible requests were ready.

The scheduler always selected the oldest ready request first. Its bucket owned that dispatch. This prevents a full short bucket from repeatedly bypassing an older long request, but can sacrifice throughput. The configured 15 milliseconds is not a bound on total waiting: the single worker can still be busy. Production needs separate admission, queue-size and deadline policies.

Measured latency, padding and memory

All three policies completed 64 of 64 requests. Mean requests per execution were 1.00, 8.00 and 2.67 respectively. Fixed batching allocated 16,384 positions; bucketing allocated 4,288; singleton execution allocated exactly 4,208. The maximum observed queue waits were 141.16, 2,047.20 and 144.19 milliseconds respectively. This illustrates why a formation-delay setting cannot be reported as an end-to-end queue guarantee.

Latency starts at the intended arrival and ends after inference and pooling. It includes local dispatch lateness and worker waiting, but excludes tokenization, network transport and HTTP handling: all tokens were prepared before replay. Each policy ran once, sequentially, on a shared host. The fixed-width result combines grouping with substantial padding; it does not isolate the causal effect of batch size. Add longest-in-batch padding without bucketing as a separate ablation before choosing a production policy.

Memory is process RSS sampled approximately every millisecond, not peak tensor allocation or GPU memory. The same process retained model and allocator state across runs. The similar fixed and bucket RSS values therefore cannot establish their independent memory footprints. Use fresh processes and device-specific instrumentation for a capacity decision.

Most importantly, bucketing did not beat the singleton p95 here. Lower padding alone is not a release criterion. Repeat trials in randomized order, retain class-specific sample counts, and evaluate the application's arrival-to-completion SLOs rather than promoting an attractive occupancy ratio.

Read the occupancy sketch correctly

This compact illustration uses four hypothetical lengths, not the measured corpus. A filled block is an attended position and a dot is padding. The same four requests keep their lengths in both layouts. Grouping similar lengths removes rectangular empty space; it also changes dispatch order and potentially waiting. The sketch cannot predict kernel time or a GPU speedup.

Restore identity and test sparse buckets

The executed bucket run changed completion order. Results were collected by request ID and rebuilt in original ID order before comparison with singleton embeddings. The maximum absolute component difference was 0.0 in this fixture for both batched policies. That is observed equality in this runtime, not a universal promise of bitwise equality across devices or kernels. Retain an explicit tolerance and retrieval-quality checks for production.

The scheduler regression test placed one older long request ahead of eight short requests and asserted that the long request was selected first. This checks the selection rule, not sustained-load fairness. Add continuous short arrivals, cancellation, expired deadlines, worker failure and a bounded drain test before deployment. Never omit unfinished long requests from latency reports; they are censored work, not fast successes.

A reproducible implementation core is: tokenize without truncation; reject unsupported lengths; retain request_id and arrival; choose the oldest admitted request; select compatible ready members; dispatch when the batch is full or its formation deadline expires; pad and construct the attention mask; execute; pool with that mask; store output by request_id; restore the caller's requested order. Measure queue age before inference and record completed, rejected, cancelled and unfinished outcomes separately.

Triton shape compatibility is a separate gate

NVIDIA's dynamic batching guide describes batching for stateless models, maximum formation delay and queue policies. It advises against setting preferred batch sizes for most models without a demonstrated performance reason. Do not copy an attractive preferred size from another deployment.

The ragged batching documentation states that ordinary dynamic batching requires matching input shapes across requests. Client-side padding or a compatible wrapper must create those shapes. Merely assigning requests to the same length bucket does not make their tensors identical in shape.

Ragged input is not a universal Transformer acceleration flag. The documented backends concatenate ragged inputs into one-dimensional tensors; the model needs compatible inputs and boundary information to interpret requests correctly. A conventional dense attention implementation will not acquire that contract simply because allow_ragged_batch is enabled. This article executed neither Triton nor a ragged backend. Validate the exported model, masks, boundaries and outputs in the actual serving path before claiming support.

Decision table: change one failure mode at a time

Release checklist and next experiment

Gate 1 — retain model, tokenizer, input and arrival-trace digests, exact dependencies, thread settings, padding/truncation policy and rollback configuration. Keep customer text out of routine metrics; record lengths and bounded aggregates rather than token IDs or documents.

Gate 2 — compare an unchanged baseline, longest-in-batch padding, fixed-width padding and bounded bucketing on the same intended arrival trace. Separate low-load, burst and near-capacity trials. Record timeouts and unfinished work with the offered-request denominator, not only successful-response percentiles.

Gate 3 — verify output identity, numerical tolerance, long-input handling and task quality. Test one sparse long bucket against continuing short arrivals. Keep oldest queue age observable even when nothing completes.

Gate 4 — require class latency limits and error budgets to pass before considering throughput or memory savings. Keep the simpler policy if extra scheduling complexity has no repeatable operational benefit. CPU evidence is not a substitute for measurements on the intended GPU, backend and precision.

For a batching review, bring the encoder manifest, a sanitized length distribution, an approved arrival trace and your per-class latency limits. An implementation assessment can turn those inputs into a reproducible experiment and a rollback gate—not a promise that larger batches will be faster.