vLLM’s automatic prefix caching hashes full KV blocks into a parent-linked chain, which makes reuse block-aligned and prefill-only, and means the hit rate is a ratio of two counters rather than a tuning knob.
Automatic prefix caching (APC) keeps the KV cache of blocks that have already been computed, so a later request sharing the same prefix skips recomputing that part of the prefill. In vLLM’s engine configuration the switch is enable_prefix_caching, and its declared default in CacheConfig is True; the server flag to turn it off is --no-enable-prefix-caching. The relevant question in production is therefore not whether to enable it — it is already on — but why the hit rate is 12% when the deployment serves one system prompt to every request.
The answer is usually in the hashing scheme. vLLM does not match prompts as strings. It hashes blocks, and a block hash is built from three inputs: the parent block’s hash value, the tuple of token IDs in the block, and extra hashes for anything that must keep the block distinct, such as LoRA IDs, multi-modality input hashes, and cache salts. The hash algorithm is selectable through prefix_caching_hash_algo, which accepts sha256, sha256_cbor, xxhash, and xxhash_cbor, with sha256 as the default.
Why reuse is block-aligned
The chain construction means every block hash encodes the entire prefix that precedes it, not just its own tokens. Two requests share a cached block only if every token before that block is identical and every token inside it is identical. A single differing token at position 3 invalidates every block after it, because each subsequent parent hash differs.
The second constraint is stated plainly in vLLM’s design documentation: only full blocks are cached. A partially filled block belongs to the request that is currently writing into it and does not enter the cache until it fills. A 900-token system prompt does not produce 900 tokens of reusable cache; it produces cache for the whole blocks it fills, and the remainder is recomputed by the next request. The practical consequence is that a prompt prefix which ends mid-block wastes the tail, and prefixes that vary in length — a timestamp or a user ID injected before the shared instructions — push the divergence into block 0 and eliminate reuse entirely.
Eviction is LRU over a free queue. When a request finishes, its freed blocks are appended to the tail of the queue; when the allocator needs a block, it pops from the head, which is the least recently used entry, removes that block’s ID from the cache, and deletes its hash. Cached blocks therefore survive exactly as long as memory pressure allows. A deployment whose KV cache is near capacity evicts the shared prefix between requests and reports a low hit rate for reasons that have nothing to do with prompt structure.
Multi-tenancy has an explicit control rather than a structural guarantee. A request may carry a cache_salt, which is injected into the hash of the first block so that only requests carrying the same salt can reuse those blocks. Without a salt, two tenants sending byte-identical prefixes share cached KV blocks, which is the intended behavior and is exactly what tenant isolation requirements sometimes forbid.
The metrics that describe it
vLLM exposes prefix cache behavior as two counters, not as a ratio. The documented names are vllm:prefix_cache_queries (number of prefix cache queries) and vllm:prefix_cache_hits (number of prefix cache hits). The older hit-rate gauge is deprecated in favor of the counter pair, precisely so the ratio can be computed over a window instead of read as a lifetime average. Counters are exposed to Prometheus with the conventional _total suffix.
rate(vllm:prefix_cache_hits_total[5m])
/
rate(vllm:prefix_cache_queries_total[5m])That query answers “what fraction of block lookups hit in the last 5 minutes,” which is the number that moves when a prompt template changes. Read alongside it:
| Metric | Type | What it tells you |
|---|---|---|
vllm:prefix_cache_queries | Counter | Block lookups attempted |
vllm:prefix_cache_hits | Counter | Block lookups served from cache |
vllm:kv_cache_usage_perc | Gauge | Fraction of KV cache blocks in use (0–1) |
vllm:kv_block_lifetime_seconds | Histogram | Time from block allocation to eviction |
The pairing matters more than either metric alone. A hit rate that falls while vllm:kv_cache_usage_perc sits near 1 is an eviction problem, addressed by reducing --max-model-len or concurrency, or by giving the engine more of the card. A hit rate that falls while KV usage is moderate is a prompt problem, addressed in the template. vllm:kv_block_lifetime_seconds separates the two directly: blocks that die seconds after allocation are being evicted, not missed.
Failure modes
The first is expecting decode savings. vLLM’s documentation states that APC reduces the time of processing the queries — the prefilling phase — and does not reduce the time of generating new tokens. A workload with 200-token prompts and 2,000-token completions is decode-bound, and a 90% prefix hit rate moves aggregate throughput very little. The metric can look excellent while the p99 latency it was meant to fix does not move.
The second is invisible cache-key divergence. Anything folded into the extra hashes splits the cache: a different LoRA adapter, a different multi-modal input, a different cache_salt. Two Deployments serving the same base model with different adapters do not share prefix cache even inside one engine, and a per-request salt applied for tenant isolation deliberately fragments the cache by tenant. The hit rate drops as a designed consequence, not as a regression.
The third is treating the flag as the fix. Because enable_prefix_caching defaults to True, an operator who “enables prefix caching” during an incident has changed nothing. The observable that distinguishes a working cache from a nominal one is the counter ratio, and it should be recorded before and after any prompt-template change.
Decision frame
The next time a serving deployment shows a disappointing prefix cache hit rate, the question is not whether APC is enabled — its default is on. It is which of three things is true: the prompt prefix diverges before the first block boundary, the KV cache is evicting shared blocks under pressure, or the workload is decode-bound and the cache was never the constraint. vllm:kv_cache_usage_perc distinguishes the second from the first, and the ratio of completion tokens to prompt tokens distinguishes the third from both. Only the first is fixed in the prompt template; the second is a memory-budget decision and the third means the effort belongs somewhere else entirely.