BLUF — operate self-hosted LLM inference against user-visible service-level objectives (SLOs), not average GPU utilisation. A busy GPU can coexist with acceptable queueing; a lightly utilised GPU can still leave users waiting behind a long prefill or a saturated KV cache. Start with p95 and p99 time to first token (TTFT), inter-token latency, queue time, end-to-end latency and error rate, segmented by request class. Then use GPU telemetry to explain a broken objective, not to define success.
This is a production operating pattern, not a claim that one threshold suits every model or workflow. vLLM exposes Prometheus histograms for TTFT, inter-token latency, queue time, prefill, decode and end-to-end latency, plus gauges for waiting requests and KV-cache usage. vLLM’s production-metrics documentation explicitly distinguishes these request-path signals. NVIDIA DCGM supplies complementary device health, configuration, job statistics and GPU metrics. Together, they make a useful question answerable: which stage violated the service promise, for which requests, and what change is safe to test?
GPU utilisation is an explanation, not a user contract
GPU utilisation is a sampled device-level signal. It does not state whether an interactive user saw the first token quickly, whether a long prompt monopolised admission capacity, whether a request waited in a scheduler queue, or whether a streaming response stalled. An average also hides the tail: one in one hundred requests can miss a p99 objective while the dashboard looks calm. A service objective must therefore be expressed at the request boundary and calculated by a defined traffic class and time window.
The distinction follows the observability model documented by vLLM: request-level metrics are the SLOs an SRE tracks, while server-level metrics explain them. For self-hosted deployments, retain the same discipline across the gateway, inference server, GPU exporter and client: every request needs a correlation ID, route/model revision, prompt-length bucket, output cap, tenant or workload class, and terminal reason. Do not attach raw prompts or personal content to broad operational metrics.
Reference request path: instrument the phases separately
Diagram — Client → API gateway → admission and queue → tokenizer/prefill → scheduler and KV-cache allocation → decode/streaming → response gateway → client. In parallel: vLLM request histograms and queue/KV gauges → Prometheus; DCGM GPU, memory, power, temperature, ECC/Xid and topology signals → exporter → Prometheus. A trace ID joins the request path to a bounded sample of traces. The operational rule is simple: preserve phase timings before aggregating them into one latency number.
Prefill consumes the input context and writes attention state into the KV cache; decode iteratively produces output tokens while repeatedly reading that state. These phases have different resource shapes. A long retrieval-augmented prompt can inflate TTFT and cache pressure without making streaming slow after the first token. A decode-heavy task can have acceptable TTFT but poor inter-token latency. Capacity changes, batching settings and request limits should be tested against both rather than inferred from an aggregate GPU graph.
Define a small SLO set before tuning
Interactive TTFT: p95 and p99 from gateway acceptance to first streamed token, segmented by prompt-length and route. This measures perceived start time.
Streaming continuity: p95 and p99 inter-token latency (or time per output token). This detects a response that begins quickly but becomes unusable.
Queue time: p95/p99 waiting duration and waiting-request depth. This differentiates admission saturation from model execution.
Completion time: p95/p99 end-to-end latency, segmented by input and output size. This remains necessary for non-streaming integrations.
Reliability: terminal outcomes, timeout rate, cancellation rate and request error rate. Define errors precisely; a client cancellation is not necessarily a server failure.
Capacity context: concurrency, prompt and generation-token distributions, KV-cache usage, preemption, GPU memory, power/thermal state and relevant DCGM health events.
Use histograms for latency distributions so percentile queries reflect the objective’s threshold. Prometheus’s histogram guidance explains that histograms retain bucketed observations and that bucket choice determines value-domain error. Put bucket boundaries around the SLO threshold and its expected failure range; a coarse set of buckets around a 2-second TTFT objective cannot reliably distinguish 1.9 from 2.4 seconds. Keep labels bounded: model revision and route are useful; request ID, user ID and prompt text are not.
Decision table: metric, likely symptom, corrective action
| Signal | Likely bottleneck | First corrective action | Trade-off |
|---|---|---|---|
| p99 TTFT rises; inter-token latency stable; queue time rises | admission saturation or long prefill contention | test a concurrency/admission limit and isolate long-context traffic | lower peak throughput or require a separate route |
| TTFT rises; queue time stable; prefill time rises with prompt length | prefill capacity or excessive context | cap/retrieve less context; test prefill/decode separation | may reduce answer context or add infrastructure complexity |
| Inter-token latency rises; TTFT acceptable; decode time rises | decode capacity, batching or clock/power constraint | test output caps, batch configuration and GPU health | can reduce response length or throughput |
| Queue time and KV-cache usage approach saturation; preemptions increase | KV-cache pressure | reduce context/concurrency, add cache capacity or route long contexts | cost, lower concurrency or routing complexity |
| Latency and error rate rise with DCGM thermal/ECC/Xid health signals | device/host fault or throttling | remove node from service; run approved diagnostic and replace/escalate | temporary capacity loss |
| GPU utilisation high but objectives pass | efficient load, not automatically a problem | retain headroom test and watch tails | additional headroom costs capacity |
Load-test the saturation curve, not one happy-path benchmark
A credible capacity decision is a matrix, not a single tokens-per-second result. Freeze model revision, tokenizer, runtime flags, hardware profile and gateway policy. Generate representative classes: short interactive questions, retrieved long-context questions, structured extraction, and long generation. For each class, sweep offered concurrency and arrival rate until the first SLO breaches. Repeat mixed traffic, because a small number of long-prefill requests can change the experience of short interactive requests. Record p50/p95/p99, error outcomes, queue depth, KV-cache use, preemptions and DCGM state at each point.
Run the matrix after any material change: model or quantisation, runtime upgrade, GPU partitioning, context limit, batching policy, adapter set, gateway timeout or routing policy. This is the inference equivalent of the versioned release evidence described in structured LLM output as an API contract: the service contract includes latency and failure behaviour, not only response syntax. Store the test inputs or a reproducible generator, seeds where applicable, configuration manifest and results.
Use an error budget to govern change
An SLO needs a decision path. For example, define a monthly objective for each traffic class: a specified proportion of accepted interactive requests must meet its TTFT and inter-token objectives, while a separate reliability objective covers server-side errors. The remaining allowed misses are the error budget. It is not permission to ignore failures; it is a shared trigger. When consumption crosses a pre-agreed threshold, freeze non-essential tuning, investigate the phase that regressed, and require a controlled canary plus a saturation test before promotion. Set the numerical objectives from business tolerance and observed distributions, not from generic values in a blog post.
Tie ownership to the layers: the product owner sets the user-visible outcome and traffic classes; platform owns runtime and node capacity; SRE owns recording rules, alerts and error-budget policy; security owns access to telemetry and prompt redaction; application owners own input and output limits. This complements the operating boundary for connected AI workflows: scale only after the service has a named boundary, evidence and a safe way to stop or roll back change.
Failure modes to rehearse
Average-latency masking: mean latency remains flat while p99 TTFT fails for long-context users. Mitigation: alert on class-specific p95/p99 and inspect histogram buckets, not only averages.
Blind cache saturation: a global GPU graph looks normal while KV-cache usage, waiting requests and preemptions climb. Mitigation: correlate cache and scheduler gauges with queue-time histograms; constrain context and concurrency before scaling blindly.
Misattributed gateway delay: only inference-server timing is observed, so DNS, auth, rate limiting or streaming proxy buffering disappears. Mitigation: measure at gateway acceptance and first byte/token, with trace correlation.
Benchmark-only capacity: the test uses fixed short prompts at one concurrency. Mitigation: replay a governed workload matrix including the long-tail context and output distributions.
Unsafe tuning during an incident: operators change batching, context or model flags together. Mitigation: one change per canary, a rollback manifest, and an error-budget freeze policy.
GPU health blindness: application metrics are treated as proof that hardware is healthy. Mitigation: retain DCGM device health, ECC/Xid and thermal/power signals and remove unhealthy nodes through a controlled procedure.
What this does not solve
SLO instrumentation does not make a model accurate, safe or legally compliant; it does not decide the right hardware, eliminate queueing at finite capacity, or make a weak workload mix representative. Detailed tracing can add overhead, and excessive metric labels can create expensive, unreliable telemetry. DCGM exposes device signals, but it does not tell you whether an application response was useful. Treat model quality evaluation, security controls, data handling and legal obligations as separate workstreams. If processing or deployment has regulatory implications, obtain legal review; the engineering pattern here is measurable service operation, not legal advice.
Start with one service, one user journey and one saturation test
Choose the highest-value interactive route. Define its request classes, gateway-based TTFT and inter-token objective, error semantics, owners and a short retention policy for non-sensitive telemetry. Export vLLM and DCGM signals, build one phase-oriented dashboard, and run the controlled saturation matrix before the next capacity purchase. An architecture review can turn that evidence into a route policy, dashboard specification, load-test harness and rollback plan sized to the actual workload.
