BLUF — lower-precision KV state is a capacity experiment, not a free speed switch. Keep model weights fixed, pin the attention backend and scale recipe, and accept FP8 only when more useful requests meet the same quality and latency contracts. If long-context evidence or numeric extraction regresses, extra admitted sequences are not usable capacity.
This guide supplies an experiment manifest, a decision table and an executable synthetic evidence fixture. It does not report GPU benchmark results. The fixture tests the evaluation plumbing; it cannot establish that a particular model or accelerator preserves accuracy. The operational starting point is the self-hosted inference SLO framework, not device utilisation alone.
Separate weights, live state and reserved memory
Weights are the persistent model parameters. KV state holds keys and values associated with the tokens retained for attention. Reducing weight precision and reducing KV precision address different parts of the memory budget. With conventional full attention, more retained tokens and concurrent sequences require more state; grouped-query attention, sliding windows and alternative attention architectures change that accounting.
For a simplified homogeneous full-attention decoder, unsharded KV payload bytes equal 2 × layers × KV heads × head dimension × retained token positions × bytes per element. The leading factor counts keys and values. Sum retained positions across sequences, accounting separately for shared blocks. This excludes scale metadata, block rounding, allocator overhead, temporary activations and replicated state. Do not apply this expression unchanged to latent-attention or hybrid models.
Changing a two-byte KV element to a one-byte element halves that payload term, not total GPU memory, not the model's parameter memory and not necessarily latency. Engines can reserve a similar memory pool while storing more tokens inside it. Therefore report both allocated device memory and the engine's usable KV-block/token capacity. A flat device-memory reading does not disprove a capacity improvement.
Read the diagram as a memory-boundary model, not a measured memory distribution. Weights stay unchanged; the candidate changes the representation of attention state. Extra room still has to pass admission, latency and answer-validation gates.
The intervention may include attention arithmetic
The vLLM quantized-cache documentation explicitly states that Flash Attention 3 with FP8 KV cache also performs attention operations in the quantized domain and quantizes queries. Calling that a storage-only experiment would hide a relevant numerical change. Record the actual backend selected at startup, not merely the requested backend.
The documentation describes default scales of 1.0 and dataset calibration through llm-compressor, including per-attention-head strategies with backend restrictions. These are version-sensitive capabilities. The linked latest documentation is a developer preview: verify flags, supported layer types and effective scale loading against the deployed release, startup logs and help output before executing the experiment. An unsupported candidate is an invalid test cell, not a failed quality score.
The vLLM engineering analysis reports kernel-accumulation fixes, head-dimension-dependent prefill behaviour and hybrid-layer exceptions. Its measurements describe its models, kernels and accelerators; they are not measurements of your endpoint. Its advice to calibrate does not guarantee that calibration improves every held-out segment. Treat calibrated and uncalibrated variants as separately evaluated candidates.
Do not transplant another library's API. Transformers cache strategies documents QuantizedCache and warns that cache quantization can hurt short-context latency when memory is sufficient. Its cache_implementation setting is not a vLLM deployment flag. The portable lesson is the memory–latency trade-off, not interchangeable configuration.
Freeze a manifest before changing precision
Record an immutable baseline B0 with image digest, vLLM version or commit, attention-kernel revision, GPU model/count, driver and runtime versions, model revision, tokenizer revision and chat-template digest. Include weight dtype and quantization recipe even when unchanged. A model directory containing new scale metadata must be compared against the original tensors so a hidden weight change does not enter the experiment.
Add KV dtype, skipped layers, scale granularity, scale source, calibration artifact digest, effective backend and query/attention precision. Record whether default, checkpoint-provided or runtime-derived scales actually loaded; do not infer this from a filename. If the release supports a runtime scale-computation path, treat it as another explicit variant rather than assuming it matches offline calibration.
Freeze tensor parallelism, context limit, memory budget, scheduler limits, prefix-cache policy, speculative decoding, decoding parameters, stopping policy and output cap. Preserve actual output lengths too: a shorter answer can look faster while being less useful. Keep calibration data separate from evaluation data, with immutable manifests and deduplication rules. Include representative languages and long-context structure in calibration without exposing protected evaluation answers.
A minimal machine-readable record uses fields baseline_id, candidate_id, image_digest, model_revision, tokenizer_revision, weights_digest, attention_backend, kernel_revision, gpu_inventory, kv_dtype, scale_granularity, scale_artifact_sha256, calibration_corpus_sha256, evaluation_corpus_sha256, skipped_layers, effective_args, trace_sha256 and evaluator_revision. Store measurement fields separately and leave them absent until runs produce results. A missing TTFT value is not zero latency.
Baseline–candidate decision table
| Test cell | Held constant | Evidence required | Decision |
|---|---|---|---|
| B0 approved precision | Full production manifest | Per-class quality and SLO baseline | Retain as rollback target |
| C1 supported FP8 default scales | Weights, requests and scheduler | Effective scales; long-context and numeric failures | Reject on any protected-segment gate breach |
| C2 supported calibrated FP8 | Same held-out evaluation and workload | Calibration provenance; actual loaded scales | Promote only if quality and goodput both qualify |
| C3 supported selective layer exclusion | Same model and test trace | Changed-layer inventory; memory and latency | Treat as a distinct intervention |
| Unsupported backend or scale recipe | No substitute runtime | Compatibility evidence missing | Mark invalid; do not manufacture a result |
The table defines experiments, not benchmark outcomes. Do not silently switch kernels between C1 and C2 and attribute everything to calibration. Where coupling cannot be avoided, name the complete intervention and compare the deployment bundles rather than claiming a causal cache-only effect.
Use paired quality fixtures before a load sweep
Begin with identical approved prompts sent to B0 and each compatible candidate. Stratify by short, medium and long tokenized input length, actual output length, German and English, evidence position and numeric task type. Place the required fact near the beginning, middle and end; include distractors with similar identifiers and deliberate missing-answer cases. Tokenize using the pinned tokenizer: repeated characters are not a trustworthy context-length measurement.
A reproducible synthetic fixture can place the record “Asset AX-104; revision B; nominal torque 12.5 N.m” among unrelated records and ask for asset, revision, nominal_torque and unit as a JSON object. The expected tuple is AX-104, B, 12.5, N.m. Reject a wrong revision, a factor-of-ten error, the wrong unit, missing fields or additional fields. Include an absent-asset request whose approved answer is a refusal rather than an invented value. This is fabricated test data, not maintenance guidance.
The companion fixture was executed locally to check correct-answer acceptance and injected-error rejection. It did not call an LLM. Passing a validator self-test establishes only that these checks work; run the actual model comparison on an isolated serving environment before making capacity claims. Extend beyond literal extraction to multi-document evidence, supported reasoning and business-rule validation, with review of paired disagreements. Keep refusals, malformed outputs and timeouts in the denominator.
Measure useful capacity, not just admitted concurrency
First compare variants at the same offered arrival trace and approved admission limit. Then sweep offered load and admission limits separately to identify each candidate's operating envelope. A closed-loop concurrency test diagnoses saturation but can hide queue growth because slow responses suppress new arrivals. Also use an open-loop trace with recorded planned arrival, actual dispatch and client-side delay.
For every context-length class record offered and completed requests, accepted answers, deadline failures, queueing, TTFT, inter-token latency and end-to-end latency. Distinguish engine token timing from network streaming-chunk gaps. Capture allocated memory, effective KV capacity, waiting requests and preemption counter deltas with restart boundaries. Inspect the deployed metrics schema before choosing names or attributing engine-wide counters to a request class.
Define useful goodput as responses passing both task validation and their class SLO divided by a fixed observation interval. Assign the cohort by arrivals within the interval, apply a predeclared drain deadline and keep unfinished work visible. Count rejected and failed requests in offered demand and operating cost. Report sample counts and repeated-run variation; an aggregate improvement can hide long-context degradation.
Randomize run order, repeat runs and separate engine warm-up from prefix-cache warm-up. Keep reuse policy fixed for the precision comparison; evaluate realistic reuse separately using the prefix-caching experiment. More resident sequences can increase queueing or preemption. If the bottleneck moves to compute, memory savings alone do not justify expanding admission.
Rollback and acceptance checklist
Check 1 — compatibility: the pinned image loads the intended model, backend, scales and layer exceptions. Archive the effective configuration and startup evidence. No silent fallback or unknown scale provenance is acceptable.
Check 2 — quality: service owners set absolute quality floors and allowable paired regressions before results are reviewed. Every protected language, long-context and numeric segment must pass. Where sample sizes cannot distinguish an acceptable change, collect more evidence rather than calling the result equivalent.
Check 3 — operations: require the agreed TTFT, streaming, error and completion contracts at the target offered load. Record the cost per accepted, SLO-compliant response, including retries. Reserve baseline capacity: a smaller KV representation does not guarantee that rollback to a larger representation fits the candidate's admission limit.
Check 4 — recovery: canary a bounded route, send new traffic back to the baseline on a breach and drain existing streams only until the declared deadline. Reduce admission before restoring the larger cache where necessary. Rehearse restart, cache loss and warm-up; do not depend on reconstructing the approved manifest during an incident.
Choose the next experiment from the failure
If weights dominate, use the separate weight-quantisation and adapter quality gates. If KV state is the constraint but protected answers regress, investigate scales, backend numerical behaviour and supported layer exclusions before purchasing the apparent concurrency gain. If quality holds but goodput does not improve, retain the simpler baseline.
For an inference architecture review, bring the immutable serving manifest, a redacted arrival trace, representative held-out requests and per-class acceptance limits. The useful deliverable is a tested retain, calibrate or roll-back decision—not a promise that FP8 doubles your serving capacity.
