BLUF — prefix caching can remove repeated prompt-processing work. It does not remove the work of generating an answer. Approve it on cost per accepted, SLO-compliant response under a representative traffic mix—not on a warm-cache time-to-first-token screenshot. For self-hosted inference, the central trade-off is reusable KV state versus effective serving capacity; for an API buyer, it is the provider’s actual cached-input billing contract. Neither is equivalent to a universal discount on tokens.
This article provides an experimental design and a transparent cost worksheet, not measured performance results. The distinct question is whether repeated prefixes justify their operational and memory footprint. Start with the self-hosted inference SLO framework for phase instrumentation; use the experiment below before approving caching as a capacity or procurement saving.
What is reused—and what is not
vLLM’s automatic prefix caching documentation describes reuse of previously computed key/value state when a new request shares an existing prefix. The engine skips computation for the reusable portion of the prompt. The same documentation explicitly limits the benefit to prefill rather than generation of new tokens. Decode still consumes the context state and produces the requested output.
A cache hit is therefore not a cached answer. Two requests can reuse a system prompt and document prefix while asking different questions and generating different answers. Conversely, semantically similar text is not necessarily a reusable prefix: token identity, order and preceding context matter. The vLLM design documentation describes hashes incorporating block tokens and preceding-prefix information, and reuse of full blocks. A shared suffix after an early changed token is not interchangeable with a shared prefix.
Place stable instructions and approved shared material before request-specific content when this preserves task semantics. Do not move security instructions, alter chat roles or weaken retrieval freshness simply to raise a hit counter. Freeze tokenizer, chat template, model and adapter revisions in the experiment. A timestamp or request identifier inserted early in the model input can destroy useful reuse; keep operational metadata outside the prompt when the model does not need it.
Separate three different cache decisions
Hugging Face’s cache-strategy guide distinguishes dynamic, static, offloaded and quantized KV-cache approaches. These describe how attention state is represented, allocated or placed. They are not, by themselves, a multi-request prefix-sharing service. The guide also shows explicit prefix prefill and reuse; application code still needs a correct lifecycle and isolation contract.
Static allocation can help compilation but reserve more capacity than short requests need. Offloading moves state between CPU and GPU and may reduce throughput. Quantized state saves memory but can add latency and needs quality evaluation. These choices can coexist with prefix reuse; benchmark them as separate changes. Do not combine a cache-strategy change, model quantisation and prefix caching into one test and attribute every difference to APC.
Keep the decoding policy fixed too. Transformers’ generation-strategy documentation explains that greedy decoding, sampling and beam search change token selection. Output caps, stopping rules and observed output lengths belong in the manifest. A shorter answer is not evidence that prefix caching accelerated decode.
Cost waterfall: an explicit accounting model
Use a work-attribution worksheet before trying to convert latency into money. For one fixed request class, define P as baseline prefill cost, D as baseline decode cost, and F as other allocated serving cost. Let h be the fraction of prefill work avoided across all requests—not merely the fraction of requests with at least one cache hit. Let H represent cache-related overhead and any separately accounted capacity penalty. The first-order model is C_base = P + D + F; C_cache = (1 − h)P + D + F + H. This assumes comparable output work and serving conditions; measure deviations under mixed load.
| Waterfall step | Illustrative cost units per request | Meaning |
|---|---|---|
| Baseline prefill | 40 | Repeated and unique input processing |
| Baseline decode | 50 | Output generation remains necessary |
| Other baseline cost | 10 | Allocated non-phase cost |
| Baseline total | 100 | Starting point |
| Avoided prefill | −30 | Assumed 75% reduction of prefill work |
| Cache overhead and capacity penalty | +5 | Explicit modelling assumption |
| Cached total | 75 | Net reduction of 25 cost units |
These are invented scenario inputs for arithmetic, not a GPU benchmark or a provider price quote. The calculation is 100 − 30 + 5 = 75: a 25% total reduction, not a 75% total reduction. Do not add a capacity penalty twice if it is already captured by a full-fleet invoice or GPU-hour denominator. The simplified break-even condition is hP > H; production approval additionally requires quality, isolation and SLO gates.
For owned or reserved GPUs, less prefill work does not automatically lower the bill. Savings become cash savings only when capacity can actually be removed or avoided; otherwise report increased SLO-compliant throughput or recovered headroom. Calculate fleet cost per accepted response using the whole test interval, including idle capacity and warm-up. Count retries and failures in costs, but not as successful useful responses. For provider APIs, reconcile billed uncached input, cached input, output and any applicable storage/write charges using the actual contract. Do not apply this worksheet as a substitute for vendor billing rules. The model-routing decision plane supplies the complementary policy and reconciliation boundary.
Benchmark matrix: cold, warm and realistic
Freeze the serving image digest, model/tokenizer revisions, hardware, parallelism, memory budget, scheduler settings, generation configuration, prompt families and arrival trace. Keep model loading and kernel warm-up separate from prefix-cache warm-up: a cold prefix cache should not mean an otherwise uninitialised runtime. Use independent runs or a verified reset method; record how cold state was established.
| Test cell | Prefix and output mix | Record in every run | Decision exposed |
|---|---|---|---|
| APC disabled baseline | Representative inputs and outputs | Hit metric definition; TTFT; inter-token latency; peak memory; throughput | Comparable reference, not an uninitialised server |
| APC enabled, cold | Same trace without seeded prefixes | Same metrics plus warm-up cost and failures | First-use penalty and readiness |
| APC enabled, warm | Repeated long prefixes, short outputs | Same metrics plus reused input-token share | Best plausible reuse case |
| Mixed production replay | Repeated and unique prefixes, original arrival gaps | Same metrics plus eviction and preemption observations | Whether reuse survives traffic churn |
| Decode-heavy stress | Same prefixes, long outputs | Same metrics plus actual output lengths | Lower ceiling on total savings |
| Isolation and restart | Separate trust groups; restart or rollout | Same metrics plus cross-group reuse tests | Security boundary and recovery cost |
This is an unfilled measurement matrix, not a set of predicted results. Report p50/p95/p99 TTFT and inter-token latency, successful requests per second, output tokens per second, actual peak device memory and engine KV-block utilisation where available. Inspect the deployed metric schema before naming a counter. Prefix-hit metrics may count tokens, blocks or requests; write down the numerator and denominator. A request with a tiny reusable prefix must not be treated as equivalent to one that reuses most of a long prompt.
Sweep offered arrival rate while preserving realistic prefix frequency and reuse distance. A closed-loop client that waits for each response can hide overload; also test the actual arrival pattern with bounded admission and record rejected or timed-out requests. Randomise run order and repeat runs so thermal state, background load and cache history do not masquerade as a treatment effect. Preserve per-class results rather than averaging short chat and long document generation together.
Memory pressure and isolation are release gates
Cached blocks occupy a managed pool, not necessarily a separately allocated second copy of the entire model context. Sharing can reduce duplicate state, while cold, low-reuse blocks can be evicted for new work. Therefore, high allocated GPU memory alone does not establish that APC is harming capacity. Inspect active versus reusable state where the engine exposes it, and correlate eviction, preemption, waiting requests and useful throughput. Reject a warm-cache optimisation that worsens the unique-prefix traffic class beyond its agreed budget.
vLLM documents cache_salt for limiting reuse to requests that share a salt. Treat it as one engine mechanism, not a complete tenant-authentication system or a general guarantee against side channels. Derive the trust-group assignment at an authenticated gateway; do not let an untrusted caller select another group’s value. Verify support and behaviour in the exact deployed version and request path. Test same-prefix requests within a group and across groups, including proxy forwarding and restarts. Do not log confidential prompts or secret salt material as metric labels.
Release checklist and rollback decision
Check 1 — evidence: capture the versioned manifest, cold-state method, traffic composition, raw aggregate measurements, denominator definitions and run variability. Separate observed results from the accounting assumptions.
Check 2 — acceptance: require no regression beyond agreed per-class TTFT, streaming, error and quality budgets; compare total cost per accepted response, not only the cache-hit ratio. Establish thresholds with the service owner before seeing candidate results.
Check 3 — security: verify trust-group isolation, approved data handling and lifecycle behaviour. If cross-boundary reuse cannot be excluded, keep reuse disabled for the affected route rather than assuming a default setting is sufficient.
Check 4 — rollout: canary one route with a documented switch back to its baseline configuration. Rehearse cold-cache behaviour during restart and rollback; a disabled or flushed cache can cause a prefill surge. Preserve enough headroom or admission control to keep that surge bounded.
Turn the result into a capacity decision
Choose one repeated-document or shared-instruction workload and one unique-prefix control. Bring a redacted arrival trace, serving manifest and cost boundary to an architecture review. The deliverable should be a reproducible benchmark, an isolation test and an explicit retain/disable decision—not a promised universal speedup. If measured prefill savings cannot survive realistic churn, output lengths and trust boundaries, spend the next optimisation cycle on the bottleneck the evidence actually identifies.
