You're in an ML Engineer interview at Anthropic.
The interviewer asks:
"Our model generates 100 tokens in 42 seconds. How do you make it 5x faster?"
You: "I'll optimize the model architecture and use a better GPU."
Interview over.
Here's what you missed:
The real bottleneck isn't compute. It's redundant computation.
Without KV caching, your model recalculates the same attention keys and values for every single token generation.
That's why a 9-second inference becomes 42 seconds. You're wasting 80% of your time on repeated calculations.
The fundamental issue:
(refer image below as you read ahead)
LLM token generation is autoregressive:
- Generate token 1 from the prompt
- Generate token 2 from prompt + token 1
- Generate token 3 from prompt + token 1 + token 2
At each step, you're reprocessing ALL previous tokens through attention.
Token 50? You've computed attention for token 1 fifty times.
The reality of attention mechanism:
For each token, the transformer computes:
- Query (Q) from current token
- Key (K) from all previous tokens
- Value (V) from all previous tokens
Then: Attention(Q, K, V) = softmax(QK^T)V
Problem: K and V for previous tokens never change. You're recalculating identical matrices every single step.
How KV caching solves this:
Instead of recomputing K and V matrices:
- Cache them after first computation
- Reuse cached values for subsequent tokens
- Only compute K and V for the new token
Without KV caching (token 50):
- Compute Q, K, V for all 50 tokens → O(n²)
With KV caching (token 50):
- Load cached K, V for tokens 1-49
- Compute Q, K, V only for token 50 → O(n)
You've eliminated quadratic redundancy.
So what's the tradeoff:
While KV caching makes the inference faster, it also takes up a lot of memory, so there is always a tradeoff between speed and memory.
Why your first token always takes longer:
KV caching speeds up inference by computing the prompt's KV cache before generating tokens.
This is exactly why ChatGPT takes longer to generate the first token than the rest.
First token: Computing KV cache for entire prompt
Remaining tokens: Just loading cached KVs + computing new token
----
Everything above is how KV caching works inside a single request.
Running it in production is a different problem. Caches break on document reordering, and a single GPU throws away roughly 15 TB of reusable cache per day.
I wrote an article that picks up exactly where this post ends: how a new open-source architecture manages KV cache at production scale, with 14x faster time-to-first-token to show for it.
The article is quoted below.