Area 05 // AI System DesignLesson 84 / 14013 min read
Token budgets and context window management
A token budget is a planned limit on how many input and output tokens each feature, request or task may use, and context window management is the system that decides what to keep, trim, summarise or retrieve so every request fits its budget and the model's window.
Think of it like this
Analogy
A family has a monthly budget: so much for rent, food, transport. When food costs rise, they don't stop paying rent; they cut snacks first. A suitcase works the same way: essentials go in first (passport, medicine), then clothes for this trip, and if there's still room, nice extras. What doesn't fit stays home or gets sent separately.
Token budgets are the family budget for model calls; context management is packing the suitcase.
The problem it solves
Without limits, prompts grow by themselves: chat history keeps getting longer, retrieval returns more chunks, tool results pile up, someone adds three paragraphs to the system prompt. Costs creep up, latency grows, requests start failing at the context limit, and answers get worse because the important parts are buried. You need explicit budgets and a system that enforces them on every request.
How it works: token budgets
- Set budgets per feature: input tokens (e.g. 6,000 for chat, 20,000 for document summary), output tokens (
max_tokens), and for agents a per-task total (stop conditions). - Set budgets per user or tenant: daily token or cost limits, to stop abuse and runaway usage (denial of wallet).
- Count before sending, with the provider's token counter or tokenizer (tokenization).
- Enforce: trim to fit, reject with a helpful message, or route to a model with a bigger window.
- Watch them: dashboards of tokens per request by feature, alerts when they drift.
How it works: context window management
- Know the window and reserve output: input budget ≤ window −
max_tokens− a safety margin (context windows). - Assemble by priority: system instructions and the current question first; essential state (the user's goal, the plan); retrieved documents (top few); recent history verbatim; older history as a summary.
- Trim with the right tool:
- Sliding window: keep the last N turns.
- Summarise older turns into a short note (compaction).
- Drop or shrink tool results once used.
- Retrieve instead of include: store long history or documents and fetch only relevant parts (RAG).
- Keep stable parts stable for caching: system prompt and tools first, changing content last.
- Test the edges: very long conversations, huge pasted documents, many tool results.
See it in code
- Needs
- Nothing: plain Python 3.
Fitting a long chat into a per-feature budget: keep the system prompt and question, then as many recent turns as fit, and mark the rest for summarising.
def tokens(text: str) -> int:
return max(1, len(text) // 4) # rough; use the provider's counter in production
BUDGETS = {"chat": {"input": 6_000, "output": 800}, "summary": {"input": 20_000, "output": 400}}
def fit_history(system: str, history: list[str], question: str, feature: str) -> list[str]:
"""Context window management: keep system + question, then as many RECENT turns as fit."""
budget = BUDGETS[feature]["input"] - tokens(system) - tokens(question)
kept = []
for turn in reversed(history): # newest first
if tokens(turn) > budget:
break
kept.insert(0, turn)
budget -= tokens(turn)
dropped = len(history) - len(kept)
summary = [f"[summary of {dropped} older turns]"] if dropped else []
return [system] + summary + kept + [question]
system = "You are a helpful assistant." * 10
history = [f"turn {i}: " + "words " * 900 for i in range(1, 11)] # ten long turns
ctx = fit_history(system, history, "And what about refunds?", "chat")
print("context parts:", [c[:22] for c in ctx])
print("input tokens :", sum(tokens(c) for c in ctx), "/", BUDGETS["chat"]["input"])
print("max_tokens :", BUDGETS["chat"]["output"], "(caps the reply)")context parts: ['You are a helpful assi', '[summary of 6 older tu', 'turn 7: words words wo', 'turn 8: words words wo', 'turn 9: words words wo', 'turn 10: words words w', 'And what about refunds']
input tokens : 5489 / 6000
max_tokens : 800 (caps the reply)In production the summary is written by a model (with the goal, constraints and decisions it must keep), and token counts come from the provider's counter, e.g. client.messages.count_tokens(...) for Claude.
A real example
Example
Taming a chatbot's token creep
A chatbot's average input grew from 3,000 to 14,000 tokens over three months: longer conversations, a system prompt that doubled in size, and retrieval returning 10 chunks "to be safe". Cost per request had quadrupled and p95 latency doubled.
The team introduced budgets: 5,000 input tokens for chat, 600 output. Context assembly kept the last 8 turns plus a summary, retrieval returned the top 4 chunks after reranking, and the system prompt was edited back down (and cached). Average input fell to 4,200 tokens, cost per request dropped by about 70%, and eval scores slightly improved thanks to less noise.
The easy way
Start here
Pick three numbers per feature: max input tokens, max_tokens for output, and how many past turns to keep. Count tokens before every call and trim the oldest turns first. Log tokens per request so you notice creep.
When you grow: the production setup
Teams implement a context-assembly layer with priorities and per-source caps, provider token counting, summarisation and retrieval for long histories, per-user and per-tenant quotas, budget checks in agent loops, caching-friendly ordering, dashboards of token composition per feature, and evals to confirm trimming doesn't hurt quality.
Trade-offs
What you gain
- Predictable cost and latency per feature.
- Requests never fail on context limits.
- Focused context often improves answer quality.
What it costs you
- Trimming can drop information the model needed.
- Summaries add calls and can lose detail.
- Budgets need tuning per feature and model.
When not to use it
- Never skip budgets entirely; even generous ones prevent runaway cost.
Common mistakes
- No output budget, so replies (or reasoning) run long and get cut off at the limit.
- Trimming the system prompt or current question instead of old history.
- Estimating tokens from characters for long or non-English text.
- Letting retrieval return "as many chunks as fit".
- No per-user limits, so one user or bot can spend thousands.
Interview questions
Try answering out loud first, then open the model answer.
How do you manage context in a long-running chat application?
Model answer
Define an input budget below the model's window minus the output budget. Assemble context by priority: system instructions and the current turn; pinned state (goals, constraints); top retrieved documents; recent turns verbatim; older turns as a running summary; drop stale tool outputs; store full history externally and retrieve relevant parts on demand. Keep stable parts first for caching, count tokens precisely, and evaluate long-conversation behaviour.
Why set token budgets, and at what levels?
Model answer
To control cost, latency and quality and to prevent abuse and runaway loops. Set them per request (input and max output tokens), per feature (different needs), per agent task (total tokens or cost), and per user or tenant per period (quotas). Enforce by trimming, rejecting or routing, and monitor drift.
Your requests are failing with context-length errors. What do you do?
Model answer
Find which component grew (history, retrieval, tool results, system prompt, attachments) from token logs; add a context-assembly step that counts tokens and trims by priority with summarisation and retrieval; reserve output tokens; cap retrieval and tool output sizes; and add alerts on input tokens per request. Optionally route rare large inputs to a model with a larger window.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Token budget
- A limit on how many tokens a request, step or whole task may use, to control cost, latency and context size.
- Context window
- The maximum number of tokens a model can consider in one request, counting both the input and the generated output.
- max_tokens
- A request setting that caps how many tokens the model may generate in its reply.
- Context compaction
- Replacing a long conversation history with a faithful summary plus recent turns, so a long-running agent stays within its context window.
- Retrieval
- Finding the pieces of stored information most relevant to a question, for example by keyword or vector search.
Before and after
Progress is saved in this browser only.