BLUF — chunked prefill is a scheduling trade-off, not a fairness guarantee. It can keep existing streams moving while a long prompt is processed in pieces; it does not automatically let a newly arrived short prompt jump ahead. Tune the token budget against separate short- and long-request contracts, preserve an unchanged baseline, and reject a throughput improvement that merely transfers waiting time to another class.
The practical decision is whether one shared vLLM engine can serve interactive and document-heavy traffic within their respective latency budgets, or whether admission control and separate capacity are necessary. This walkthrough supplies an explicit parameter-sweep specification, measurement contract and rollback manifest. It does not report measured GPU speedups: the publishing environment has no configured vLLM runtime or inference endpoint. The numerical settings below are experiment inputs, not production recommendations.
Understand what the scheduler actually protects
The vLLM optimisation guide documents that V1 enables chunked prefill whenever possible. With the feature enabled, pending decodes are scheduled before prefills; the remaining token budget can process a prefill or a portion of one. Prefill builds attention state for input tokens. Decode extends an already admitted sequence. The two phases compete for the same engine iteration but have different compute and memory behaviour.
That distinction matters for a short chat request arriving behind a long document. Before its first token, the chat request is also waiting for prefill. Protecting ongoing decodes is not equivalent to prioritising that new request. A shorter chunk can reduce the duration of an iteration without changing which waiting prefill receives the next opportunity. Do not label a decode-first schedule “shortest job first”.
The guide describes smaller max_num_batched_tokens values as favourable to inter-token latency and larger values as favourable to prefill progress and TTFT. Those are tuning directions, not guaranteed results at arbitrary load. A smaller budget can require more iterations to complete a long prompt. A larger budget can extend an iteration and delay the next streamed token. Queueing, kernel efficiency, cache state and admission limits can reverse a simple single-request observation.
The linked pages are developer-preview documentation, inspected for this article. Before applying a flag, compare the deployed release documentation, vllm serve --help and the effective startup configuration. Do not import a newer option or an old partial-prefill flag from a blog into an unverified serving image.
A scheduling timeline, not a speedup chart
Read the timeline as an illustrative budget account: an iteration first reserves capacity for eligible decode work, then assigns remaining capacity to prefill work. An unfinished long prefill returns for later work; a newly arrived short request still needs admission and a scheduling opportunity. The arrows do not imply simultaneous execution on independent GPUs, fixed-duration iterations or a measured ordering of every request.
The self-hosted inference SLO framework defines the user-facing measurement boundary. This article adds the scheduler experiment: hold the service contract constant while changing only the batch-token budget and engine sequence limit. It is not another reason to maximise GPU utilisation.
Freeze a workload that can expose unfairness
Use two labelled input classes. A concrete starting specification is 512 input tokens for the short class and 8,192 for the long class, with 128 output tokens in both. Tokenise with the pinned model tokenizer; character count is not token count. Keep the prompt corpus, ordering, model revision, tokenizer revision, chat template, precision, accelerator model and count, parallelism, memory allocation and generation configuration fixed across candidates. Use an approved representative corpus, not customer secrets in benchmark logs.
For a scheduler microbenchmark, disable prefix reuse and speculative decoding or hold their verified settings fixed. Otherwise cached long prompts or changing draft acceptance can mask contention. Run a separate realistic-cache experiment later; prefix-caching economics explains why skipped prefill work must be distinguished from faster scheduling of actual work.
A fixed output cap does not guarantee a fixed output length. In a compatible synthetic benchmark you may enforce an exact generation length using supported generation controls, clearly marking it as synthetic. In a realistic replay, retain actual completion lengths and stop reasons, and compare equivalent length strata. Do not claim a scheduling win because a candidate emitted fewer tokens.
Use an open-loop arrival trace with timestamps independent of completion. Fix the same trace for each candidate. A closed-loop concurrency test is useful for saturation diagnosis, but it reduces offered work when responses slow and can conceal growing queues. Record planned arrival, actual dispatch and any client-side admission delay. If the load generator cannot keep up, invalidate that run rather than attributing its timing to the server.
Parameter-sweep artifact: explicit cells and unchanged controls
Start with the current approved configuration as B0, not a guessed library default. Then cross token budgets 2,048, 4,096 and 8,192 with sequence limits 16 and 32. These example values need memory and model-compatibility checks before use. The six candidate names below correspond to distinct server configurations, not client concurrency limits.
| Run cell | max_num_batched_tokens | max_num_seqs | Required comparison |
|---|---|---|---|
| B0 | Captured production value | Captured production value | Unchanged rollback baseline |
| C1 | 2048 | 16 | Small budget; lower sequence limit |
| C2 | 2048 | 32 | Same budget; higher sequence limit |
| C3 | 4096 | 16 | Mid budget; lower sequence limit |
| C4 | 4096 | 32 | Same budget; higher sequence limit |
| C5 | 8192 | 16 | Large budget; lower sequence limit |
| C6 | 8192 | 32 | Same budget; higher sequence limit |
The executable launch template for each candidate is: vllm serve "$MODEL" --revision "$REVISION" --tokenizer-revision "$TOKENIZER_REVISION" --enable-chunked-prefill --max-num-batched-tokens "$B" --max-num-seqs "$S". Add the exact frozen baseline arguments for precision, parallelism, context length, memory and generation; the template is not a complete deployment manifest. Restart on isolated test capacity, confirm the effective values in startup output and wait for model and kernel warm-up before sending the frozen arrival trace. Do not repeatedly restart a production service to run this sweep.
For each server cell, replay short-only, an 80/20 short/long mix and a 50/50 mix by request count. Choose a low offered rate and a near-capacity rate using B0, then freeze those rates for all candidates; do not independently tune each candidate's offered rate until it looks healthy. Replay the same traces in three randomised repeat rounds. Separate model warm-up, measurement and bounded drain intervals. This gives 126 planned runs across the seven server cells, three mixtures, two rates and three repeats. It is a test plan, not 126 executed GPU measurements.
A machine-readable sweep record should contain run_id, cell_id, repeat, trace_sha256, corpus_sha256, image_digest, model_revision, tokenizer_revision, hardware_id, effective_args, warmup_seconds, observation_seconds and drain_deadline_seconds. Store measurements separately from configuration. Leave TTFT and throughput fields absent until an actual run produces them; zero is a result, not a substitute for missing evidence.
Measure each class without inventing labels
The production metrics reference lists TTFT, inter-token latency, request queue time, waiting requests, KV-cache usage and preemption metrics. Inspect the deployed /metrics output first. Do not assume the engine attaches a short/long class label to every histogram. Capture the class at your gateway or benchmark client and correlate with approved request-level telemetry where supported; avoid prompt text, tenant identities or unbounded request IDs as Prometheus labels.
Report p50/p95/p99 TTFT and end-to-end latency separately by class, plus sample counts, errors, timeouts and cancellations. Treat server queue time as one component: client-observed TTFT can also include transport, gateway waiting, tokenisation and prefill. Current age of the oldest waiting request needs enqueue-time instrumentation; a histogram of completed queue waits will not reveal a request that never leaves the queue.
For streaming, distinguish per-token engine latency from arrival gaps between network chunks. A chunk can contain multiple tokens or be buffered by a proxy. Report client chunk-gap latency as such; use actual token timing only when the instrumentation provides it. Record per-request maximum stalls as well as pooled percentiles so long responses do not dominate the entire view.
Define class goodput as responses that pass the task validator and all class latency limits, divided by the fixed observation duration. Assign the cohort by arrivals inside that window; include rejected requests in the offered denominator. Keep unfinished requests visible and apply the predetermined drain deadline. Report accepted-over-offered fraction alongside goodput and late completions. Dropping timed-out or still-waiting requests makes an overloaded engine look artificially fast.
Take before/after preemption-counter readings with restart boundaries identified. A counter delta is engine-wide unless request attribution is verified. Correlate it with class outcomes; do not assign every preemption to the long class just because long prompts exist. Rising recomputation can erase any apparent gain from admitting more simultaneous sequences.
Decision table: which failure should change the configuration?
| Observed outcome | Check first | Candidate action | Rejection condition |
|---|---|---|---|
| Streaming stalls; prefill advances | Token timing versus proxy buffering | Test a lower token budget | Long-class TTFT or goodput breaches its contract |
| New short requests wait; streams are healthy | Admission order and oldest queue age | Bound ingress; test verified priority or isolation | A lower chunk budget alone is presented as fairness |
| Preemptions rise with sequence limit | KV pressure and counter reset boundaries | Test fewer simultaneous sequences | Memory fits only by repeatedly recomputing work |
| Aggregate throughput rises; one class loses | Per-class offered counts and failures | Retain B0 or change the capacity boundary | Any hard class gate fails |
| Long requests never complete under stress | Censored waits and admission policy | Bound waiting and reserve suitable capacity | Successful-request percentiles hide starvation |
| Both classes meet gates without a material gain | Run variability and operating cost | Keep the simpler baseline | Extra policy complexity has no measured benefit |
Isolation and priority have costs too
The engine arguments reference documents FCFS and priority scheduling; lower numeric priority is handled earlier and arrival time breaks ties. Verify support through the actual API route and exact version. A priority field that a gateway drops is not a policy. Let an authenticated gateway assign priority from approved service classes, rather than trusting callers to declare themselves urgent.
Priority alone does not promise bounded waiting or starvation prevention. Re-run a sustained high-priority flood and inspect the oldest lower-priority wait, deadline expirations and rejected work. An explicit queue-age cap or reserved capacity may be necessary. Separate deployments give a clearer capacity boundary but can strand idle capacity and add replicas, routing, monitoring and rollout work. Routing two aliases into the same contended engine is not resource isolation.
Use the model-routing decision plane to preserve approved model and data-handling constraints when traffic moves. A scheduler incident does not authorise a lower-quality model or a different data region. Admission limits should return an intentional failure contract; unbounded automatic retries merely move the overload upstream.
Rollback manifest and acceptance checklist
Store a rollback record with baseline image digest, model and tokenizer revisions, complete effective arguments, deployment revision, route target, health probe, capacity allocation, drain deadline, abort thresholds and owner. Candidate settings are an overlay on that immutable record, not a replacement assembled from memory. Pin cache and speculation settings explicitly; a restart with new defaults is not a rollback.
Check 1 — validity: all planned cells use the same approved corpus, arrival traces, hardware and generation settings. Record omitted or incompatible cells as invalid, not as passes. Confirm client dispatch lag and measurement gaps stay within the predefined tolerance.
Check 2 — acceptance: both request classes meet their TTFT, streaming, completion, error and quality contracts at the agreed offered load. Compare repeated-run variation, not only the fastest run. An improvement in one class cannot silently relax another class's hard limit.
Check 3 — failure: test a burst of long prompts, sustained decode pressure, an input at the supported context boundary, client cancellation and KV pressure. Require bounded admission and visible unfinished work. Do not infer fairness from a finite sample with no recorded queue ages.
Check 4 — recovery: canary the candidate behind a reversible route, retain baseline capacity, and rehearse routing new work back while draining existing streams for a bounded period. Define how remaining streams terminate after the deadline. Warm-up and cache loss can cause a prefill surge, so rollback capacity must be real rather than merely documented.
For an architecture review, bring the redacted arrival trace, effective serving manifest and separate short/long latency contracts. The deliverable is a reproducible retain, retune or isolate decision. Until that evidence exists, keep the baseline: chunking work more finely is not proof that the service has become fairer.
