LLM Load Tests: Expose Coordinated Omission Before Trusting Tail Latency

BLUF — an LLM load test can look healthy because its client stops offering work when the server slows. Before trusting tail latency, verify the arrival process, the measurement clock and the denominator of every reported result. Keep planned arrivals, actual dispatches, admissions, completions and missing outcomes separate. A successful-request percentile cannot describe work that was never sent.

The buying decision is whether a serving deployment can handle the intended workload, not whether a concurrency-limited script can finish its own requests. This article audits the generator rather than tuning the inference engine. It includes an executed local HTTP teaching experiment, a measurement contract and a release checklist. The experiment uses synthetic delays, not a GPU or a language model; its numbers are not inference-capacity estimates.

The feedback loop that hides overload

A closed-loop client waits for a response before sending the next request from that worker. With a fixed worker population, longer responses reduce subsequent arrivals. That is appropriate for some bounded user journeys: a user really may wait for an answer before asking again. It is misleading when used to represent an independent stream of new users or jobs that keeps arriving during a slowdown.

The wrk2 methodology explains coordinated omission and measures response latency against intended send times, rather than only actual sends. Its constant-throughput approach is a useful conceptual reference. It is not a ready-made LLM token-stream benchmark, and compensating a histogram does not reconstruct rejected requests, missing arrivals or output quality.

Our inference SLO framework defines service objectives. Here the additional question is whether the test actually exercises those objectives at the promised arrival workload. An accurate clock attached to an unrepresentative traffic process still produces an unrepresentative result.

Keep the schedule outside the response loop

The timeline is a conceptual comparison, not a plot of measured token timings. A closed worker's next arrival depends on its previous completion. An open schedule assigns arrivals independently; each scheduled item must become a dispatched request or an explicit unsent outcome. It must not quietly disappear because a concurrency semaphore was full.

Grafana's arrival-rate allocation documentation makes the generator constraint explicit: an iteration requires a free virtual user, and insufficient allocation produces dropped_iterations. Iteration rate is request rate only when the iteration's request structure makes that true. Adding virtual users dynamically also has CPU and memory costs that can perturb the test.

Choose the arrival model from the business journey. Independent job arrivals justify an open schedule; bounded conversational users may justify a closed or session-aware model with think time. Run both when both exist. Neither label makes a test representative without prompt lengths, output lengths, retries, caching and workload classes matching the decision.

An executed slow-endpoint experiment

The local Python standard-library fixture runs a real loopback HTTP server and asynchronous clients. The server has two service slots and admits at most eight outstanding requests, including executing work. Additional requests receive HTTP 503. At service start, requests normally sleep for 40 milliseconds; starts between one and three seconds into the run sleep for 400 milliseconds. A first body byte follows the delay, and a second byte follows 10 milliseconds later. These bytes are synthetic content markers, not model tokens.

Each run has a four-second arrival window and a maximum two-second drain. The open schedule plans 160 arrivals at 40 per second. The closed case has two sequential workers and no independent arrival target. The third case keeps the same open schedule but allows only two client requests in flight, recording excess arrivals as unsent_client_cap. The fixture stores one event ledger per run and asserts that no ledger item remains pending after drain.

These are actual counts from one local execution, not simulated production results. During the slowdown interval the closed run dispatched 10 requests, while the adequately provisioned open run dispatched 80. The restricted open client also dispatched only 10 in that interval, but its ledger retained the suppressed arrivals. Exact counts vary with host scheduling. The closed run's zero rejects do not prove equivalent capacity: it offered a different workload.

A teaching latency gate of 250 milliseconds from planned arrival to completion admitted 76, 75 and 50 completed responses respectively. Dividing by the fixed four-second arrival window gives 19.0, 18.75 and 12.5 latency-qualified responses per second. This is cohort-based latency-qualified throughput, including drain completions, not full LLM goodput: there is no task-quality validator. No requests were cancelled in this execution. The cancellation path exists but this run does not validate it, and no p99 claim is made from these short samples.

To reproduce the method, implement the server limits and delay schedule above; start a fresh server for each case; create open arrivals from an immutable index/rate schedule; record a row even when a client slot is unavailable; and drain to a fixed deadline. Retain the full ledger rather than just these counts. The executed fixture and ledger are retained with this article's engineering evidence; the specification here is intentionally independent of an inference vendor.

Define the ledger before calculating percentiles

Use a monotonic clock for elapsed durations. Store planned arrival, dispatch, admission, first usable content, completion or cancellation, and terminal outcome. The local fixture can share one clock because client and server run in the same process. Across hosts, do not subtract unrelated monotonic clocks: use local durations plus correlation IDs, or a documented clock-synchronisation/error budget for cross-host timestamp comparisons.

Admission is a separate server or gateway event, not automatically the client send time or receipt of HTTP headers. If the server does not expose it, mark it unavailable. Do not manufacture a queue-time decomposition by subtracting incompatible telemetry. For a real streaming API, define whether TTFT starts at client scheduling, dispatch, gateway acceptance or another boundary; publish that boundary next to the value.

Keep unsent, rejected, transport-failed, cancelled and unfinished requests outside successful-latency distributions but inside the outcome ledger and relevant denominator. At the observation boundary report outstanding requests and their current ages; at drain expiry report censored work. A timeout gives a lower bound on waiting, not a measured successful completion. Rejections have their own response latency but must not be pooled with successful generation to make the service look fast.

Pin a streaming tool and verify its traffic semantics

For an LLM run, the inspected AIPerf release is v0.13.0. Its versioned arrival-pattern documentation supports request-rate mode and an explicit constant arrival pattern; the documented default is Poisson. Its rate-plus-concurrency contract describes sleep-then-gate behaviour and no catch-up. A configured rate plus a restrictive concurrency cap is therefore not proof that all intended requests reached the endpoint.

A command template for an isolated, already-running compatible chat endpoint is: aiperf profile --model "$MODEL" --url "$ENDPOINT" --endpoint-type chat --streaming --request-rate 40 --arrival-pattern constant --benchmark-duration 60 --output-artifact-dir results/constant. Install the pinned aiperf==0.13.0 package in a suitable isolated environment and inspect aiperf profile --help before use. This template was checked against the versioned documentation, not executed against a model here. It is incomplete as an acceptance benchmark until the workload, warm-up, timeout and drain contracts are fixed.

Freeze the serving image digest, model and tokenizer revisions, prompt corpus, token-length distribution, chat template, output limits, cache policy, streaming parser and request-class mix. An output cap does not ensure identical generated lengths. A network chunk is not necessarily one token, and first-byte receipt may be only headers or an empty protocol event. Validate the parser with known streaming fixtures before calling a measurement TTFT.

Preserve per-request records or equivalent joined evidence sufficient to reconcile the schedule with dispatch and outcomes. Do not assume a tool export contains every timestamp in this article's ledger; inspect its schema. If scheduled-but-unsent work cannot be accounted for, that run cannot establish offered-load capacity. A generator CPU or connection-pool bottleneck is a test-validity finding, not a serving-performance conclusion.

A decision table for accepting or rejecting the benchmark

Only after the generator passes should you tune mixed-load prefill scheduling. Otherwise a scheduler change and a weakened offered workload can be mistaken for the same improvement. Keep the model-routing quality boundary unchanged: substituting a weaker model is not a latency-only optimisation.

Streaming benchmark acceptance checklist

Check 1 — workload: name the user journey, arrival process, request classes, input/output distribution, cache state and retry policy. Freeze the comparison manifest and record any unsupported case as invalid, not passing.

Check 2 — generator: reconcile offered, dispatched and unsent counts; inspect dispatch lag, client CPU, connection limits and protocol errors. Define tolerances before the run. Keep both quiet-period and injected-slowdown evidence.

Check 3 — outcomes: publish accepted/offered and completed/accepted fractions alongside latency distributions and their sample counts. Define goodput as task-valid responses meeting all class latency gates divided by the fixed observation window, with an explicit arrival-cohort and drain policy. Report retries as additional attempts without counting them as new user demand.

Check 4 — streaming: distinguish HTTP headers, usable content, token events and network chunks. Preserve parser version, cancellation semantics and maximum stall evidence. A fast first token does not guarantee usable completion.

Check 5 — release: repeat runs, show variability, rehearse timeout and drain behaviour, and retain a baseline rollback configuration. The short teaching fixture demonstrates omission; it does not justify production tail percentiles, a GPU purchase or a throughput guarantee.

For a benchmark-validity review, bring the serving manifest, a redacted workload trace and the generator's raw timing schema. The useful deliverable is a defensible accept, retest or resize decision—before infrastructure spend is justified by a percentile that never saw the intended workload.