Area 01 // LLMsLesson 18 / 14015 min read
Context windows and the KV cache
The context window is the maximum number of tokens a model can look at in one request (your input plus its output), and the KV cache is the memory of already-processed tokens that lets it generate each new token quickly instead of re-reading everything.
Think of it like this
Analogy
A context window is the size of a desk. Everything the model can use for this answer must be spread out on the desk at the same time: your instructions, the chat so far, any documents, and the space for the answer it's writing. Anything not on the desk doesn't exist for it, and when the desk is full something has to come off.
The KV cache is a set of sticky notes. As the model reads each word, it writes a short note about it. When writing the next word, it glances at the notes instead of re-reading the whole stack of paper from the start.
The problem it solves
Two practical problems hit every LLM app:
- Limits. Models can only look at a fixed number of tokens per request. Long chats, big documents and many tool results eventually don't fit, and the API rejects the request or your app silently drops information.
- Speed and memory. A model writes one token at a time. Naively, each new token would require re-processing the entire text so far, making long answers painfully slow. The trick that avoids this, the KV cache, uses a lot of GPU memory, which shapes how many users a server can handle.
How it works: context windows
- One request = one context. The context window is the maximum number of tokens the model can attend to in a single request, counting both input and output.
- The model has no memory between requests. "Memory" in a chat app means the app resends previous messages every time.
- Everything you send uses tokens: system prompt, conversation history, retrieved documents, tool definitions, tool results, images.
- Output is reserved inside the same window: set
max_tokens(max tokens) to cap the answer; input + max output must fit. - Sizes vary by model. As of November 2026, current flagship models commonly offer 1M tokens (Claude Opus 5.5 and Sonnet 5.5, OpenAI's GPT-6 family at about 1.05M) while many open models offer 128K–1M. Check each model card.
- Bigger isn't free: more tokens = higher cost and latency, and models often use facts in the middle of a long context less reliably than facts at the start or end (lost in the middle).
- When it doesn't fit, apps trim old messages, summarise them (context compaction), or retrieve only the relevant pieces (RAG).
How it works: the KV cache
- Recall attention (lesson 14): each token makes a query, key and value in every layer.
- Keys and values of earlier tokens never change once computed (the causal mask means later tokens can't affect them).
- So store them. The KV cache keeps every token's keys and values for every layer.
- Prefill: the whole prompt is processed in one parallel pass, filling the cache. This is compute-heavy and determines time to first token.
- Decode: for each new token, compute only its query, key and value, attend to the cached keys and values, append its own to the cache, repeat. Each step is cheap in compute but must read the weights and the whole cache from memory.
- The cost: cache size = 2 (keys and values) × layers × KV heads × head size × bytes × tokens. It grows linearly with context length and with the number of simultaneous requests, which is why serving engines manage it carefully (PagedAttention).
See it in code
- Needs
- Nothing: plain Python 3.
How much work the KV cache saves, and how much memory it costs (example model shape: 32 layers, 8 key/value heads of size 128, 16-bit numbers; similar to many 8B-class models).
# Why the KV cache matters: count attention work when generating 1,000 tokens.
prompt_len, new_tokens = 2_000, 1_000
without_cache = 0
for step in range(new_tokens):
seq = prompt_len + step + 1
without_cache += seq * seq # recompute keys/values (and attention) for every token
with_cache = 0
for step in range(new_tokens):
seq = prompt_len + step + 1
with_cache += seq # only the new token's query looks at stored keys
print(f"without KV cache: {without_cache:,} units of work")
print(f"with KV cache : {with_cache:,} units of work")
print(f"speed-up : ~{without_cache // with_cache:,}x")
# KV cache memory for one request (example: 32 layers, 8 KV heads of size 128, 16-bit numbers)
layers, kv_heads, head_dim, bytes_per = 32, 8, 128, 2
per_token = 2 * layers * kv_heads * head_dim * bytes_per # 2 = one key + one value
for ctx in (8_000, 128_000):
print(f"KV cache for {ctx:>7,} tokens: {per_token * ctx / 1e9:.1f} GB")without KV cache: 6,335,833,500 units of work
with KV cache : 2,500,500 units of work
speed-up : ~2,533x
KV cache for 8,000 tokens: 1.0 GB
KV cache for 128,000 tokens: 16.8 GBOne 128K-token conversation needs about 17 GB of cache on top of the model weights. Ten such users at once would need 168 GB. That's why long contexts are priced higher and why serving engines fight for every gigabyte.
A real example
Example
A customer-support assistant that slowly gets worse
A support assistant sends the full chat history on every turn. Short chats work well. In long chats (an hour of back-and-forth plus pasted logs), three things happen: answers get slower (more prefill), costs climb (every turn re-sends everything), and the assistant ignores an instruction the user gave early on (lost in the middle).
The fix: keep the system prompt and the last 10 turns verbatim, replace older turns with a running summary, store pasted logs outside the chat and retrieve only the relevant lines, and turn on prompt caching for the stable start of the prompt. Cost per long chat dropped by more than half and the ignored-instruction bug disappeared.
The easy way
Start here
Rules of thumb: count your tokens, keep prompts as short as the task allows, put key instructions at the start (and repeat critical ones at the end), and reserve room for the answer with max_tokens. You don't manage the KV cache when using an API; you just benefit from it, and from provider prompt caching.
When you grow: the production setup
Production systems have a context budget per feature, a trimming and summarisation policy, retrieval instead of stuffing, and monitoring of input tokens per request. Self-hosted serving tunes the KV cache directly: page-based allocation, cache quantization, prefix sharing between requests, and admission control when memory runs out (AI system design).
Trade-offs
What you gain
- Large windows let models use whole documents, long chats and many tool results at once.
- The KV cache makes generation thousands of times faster than recomputing.
- Providers' huge windows make prototyping easy: just paste everything in.
What it costs you
- More context = more cost and latency on every request.
- Quality can drop with very long, noisy contexts (lost in the middle).
- The KV cache uses a lot of GPU memory, limiting concurrent users when self-hosting.
When not to use it
- Don't fill the window just because it's there; retrieve or summarise instead when most of the content is irrelevant to the question.
Common mistakes
- Thinking the model remembers previous requests. It only sees what you send this time.
- Forgetting that output tokens share the window, then getting truncated answers.
- Stuffing whole databases into the prompt "because it fits" and paying for it on every call.
- Putting the key instruction in the middle of a long prompt.
- Sizing GPUs for model weights only and forgetting KV cache memory per user.
Interview questions
Try answering out loud first, then open the model answer.
What is a context window and how do you handle conversations that exceed it?
Model answer
It's the maximum number of tokens (input plus output) a model can process in one request. Since models are stateless, the app resends history each turn. When it grows too long: keep the system prompt and recent turns verbatim, summarise older turns, move large artefacts (documents, logs) into retrieval, drop low-value tool outputs, and track token counts so you trim before hitting the limit rather than after an error.
Explain the KV cache and its trade-offs.
Model answer
During autoregressive generation, each layer's keys and values for past tokens don't change, so they're cached. Each new token only computes its own Q, K, V and attends to the cache, turning quadratic recomputation into linear work per step. The cost is memory: proportional to layers × KV heads × head dimension × tokens × concurrent requests. That memory often limits batch size and context length, motivating GQA/MQA, KV quantization, paged allocation and prefix sharing.
What's the difference between prefill and decode, and why does it matter for latency?
Model answer
Prefill processes the entire prompt in parallel to build the KV cache; it's compute-bound and dominates time to first token for long prompts. Decode generates tokens one at a time, each requiring a read of model weights and the KV cache; it's memory-bandwidth-bound and dominates total time for long outputs. Optimising TTFT means shorter or cached prompts; optimising throughput means batching decode steps across many requests.
Quiz
Pick an answer to see whether it’s right and why.
0 of 6 answered
Key terms
- Context window
- The maximum number of tokens a model can consider in one request, counting both the input and the generated output.
- KV cache
- Stored keys and values of already-processed tokens, so each new token can be generated without recomputing everything before it.
- max_tokens
- A request setting that caps how many tokens the model may generate in its reply.
- Lost in the middle
- The tendency of models to use information at the start and end of a long context better than information in the middle.
- Prefill
- The first phase of generation, where the whole prompt is processed in parallel and the KV cache is filled. Drives time to first token.
- Decode phase
- The phase of generation where tokens are produced one at a time, each reusing the KV cache. Limited by memory bandwidth.
Before and after
Progress is saved in this browser only.