0.5 MB per Token: How the KV Cache Makes LLM Inference Possible

An LLM writes one token at a time. To choose token 2,001, attention has to look back at all 2,000 tokens before it. Done naively, that means reprocessing the entire sequence at every step - the work grows quadratically, and generation slows to a crawl within a page of text. Every production inference system avoids this with one data structure: the KV cache.

What attention does, in sixty seconds

Inside every transformer layer, each token is turned into three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what will I contribute?). When the model generates a new token, that token’s query is scored against the keys of every token before it; the scores decide how much of each token’s value flows into the new token’s representation. That weighted blend is how the model “reads” its context.

Here is the property that makes caching possible: generation is causal, left to right. A past token’s key and value never change once computed. Compute them once, and they are valid forever.

The cache stores the work, not the words

A common mental model is that the KV cache holds the previous words. It doesn’t - it holds the key and value vectors of every earlier token, for every layer and every attention head. In other words, all the processing the model has already done on the past, saved so it is never redone. Each new token computes only its own query, key, and value, then reads everything else straight out of GPU memory. Zero tokens are recomputed.

View full-size diagram ↗

Prefill and decode: one request, two different workloads

• Prefill processes your entire prompt in a single parallel pass and fills the cache. The GPU chews through huge matrix multiplications - this phase is compute-bound, and it is what you experience as time-to-first-token.
• Decode then generates one token per step. The math per step is tiny, but every step must stream the entire cache out of GPU memory - this phase is memory-bandwidth-bound, and it is what you experience as tokens-per-second.

One inference request is really two different performance problems, which is why serving frameworks schedule the phases differently - some deployments even run them on separate GPUs.

The bill arrives as memory

The cache grows linearly with context, and the constant is brutal. Take a 7B-class model: 32 layers x 32 heads x 128-dimensional heads x 2 (a key and a value) x 2 bytes in fp16 is roughly 0.5 MB per token. An 8K-token conversation holds about 4 GB of cache - for one request. Modern models shrink the constant with grouped-query attention, but the shape of the problem stays: GPU memory, not compute, caps how many conversations you can batch together, and batch size is throughput.

How vLLM serves thousands of caches at once

Since the cache is the bottleneck, serving frameworks compete on how well they manage it. vLLM’s core idea, PagedAttention, is borrowed straight from operating systems: instead of reserving one big contiguous slab of GPU memory per request (mostly empty until the conversation grows into it), it stores the cache in small fixed-size blocks - pages - allocated on demand and stitched together by a lookup table. Fragmentation disappears (the vLLM paper measured memory waste dropping from as high as 60-80% in prior systems to under 4%), and identical prompt prefixes - say, one system prompt shared by thousands of requests - can point at the same physical blocks instead of duplicating them.

The takeaway

1. LLM inference is a memory problem wearing a compute costume. Size your serving fleet by memory capacity and bandwidth per concurrent conversation, not by FLOPs - the cache decides your batch size, and batch size decides your cost per token.
2. The best new systems are old systems ideas, reapplied. vLLM’s breakthrough is virtual memory paging from the 1960s. Engineers with deep classical-systems fundamentals are exactly the ones who will keep finding these wins in the AI era.