Area 05 // AI System DesignLesson 89 / 14014 min read
Cost optimization
LLM cost is roughly requests × tokens × price per token, so you cut it by sending fewer tokens (trimming, caching), paying less per token (smaller models, routing, batch discounts) and making fewer calls (semantic caching, fewer agent steps), always checking with evals that quality holds.
Think of it like this
Analogy
Cutting a household's electricity bill: switch off lights in empty rooms (don't send tokens you don't need), use cheaper night-time electricity for the washing (batch API), buy efficient bulbs for rooms used all day (small models for easy work), and don't boil the kettle twice (caching). First, though, you read the meter so you know which appliance costs the most.
The problem it solves
An AI feature that cost a few hundred dollars in testing can cost tens of thousands a month in production. Long prompts are resent on every request, the biggest model is used for everything, agents take many steps, and nobody knows which feature or customer is expensive. You need to measure, then pull the right levers without quietly damaging quality.
How it works, step by step
- Measure first: log tokens and cost per request, per feature, per user and tenant (latency and cost metrics). Work out cost per successful task, not per call.
- Cache the stable prefix: put the system prompt, tools and fixed documents first and use prompt caching; cached input tokens are billed at a fraction of the price.
- Send fewer tokens: trim history, return fewer, better chunks after reranking, shorten the system prompt, cap
max_tokens(token budgets). - Route by difficulty: small models for easy requests, large models for hard ones (routing); check each route with evals.
- Skip calls entirely: exact and semantic caching for repeated questions.
- Use discounts: batch APIs for non-urgent work, committed-use pricing at high volume.
- Shrink agent loops: better tools, fewer steps, step and cost limits (stop conditions).
- Consider self-hosting or fine-tuning at high, steady volume: a distilled small model or an open model on your own GPUs (vLLM).
- Guard quality: run the eval set after every change; keep a cost dashboard with alerts.
See it in code
- Needs
- Nothing: plain Python 3. Example prices as of November 2026 (mid-tier $2/$10, small $0.10/$0.50 per 1M tokens, cached input at 10%).
The monthly cost of one feature (3 million requests), with each optimisation applied on top of the previous one.
# Monthly cost of a feature, then the effect of each optimisation applied in order (example numbers).
requests = 3_000_000
base = {"inp": 5_000, "out": 400, "price_in": 2.00, "price_out": 10.00}
def monthly(inp, out, price_in, price_out, share=1.0):
return requests * share * (inp / 1e6 * price_in + out / 1e6 * price_out)
steps = [("starting point", monthly(**base))]
cached = monthly(1_000, 400, 2.00, 10.00) + monthly(4_000, 0, 0.20, 0) # 4K prefix cached at 10%
steps.append(("prompt caching", cached))
trimmed = monthly(700, 250, 2.00, 10.00) + monthly(4_000, 0, 0.20, 0) # shorter history, max_tokens
steps.append(("trim context + cap output", trimmed))
routed = 0.3 * trimmed + 0.7 * (monthly(700, 250, 0.10, 0.50) + monthly(4_000, 0, 0.01, 0))
steps.append(("route 70% to a small model", routed))
steps.append(("semantic cache 20% hit rate", routed * 0.8))
for name, cost in steps:
print(f"{name:28} ${cost:>9,.0f}/month")
print(f"total saving: {1 - steps[-1][1] / steps[0][1]:.0%}")starting point $ 42,000/month
prompt caching $ 20,400/month
trim context + cap output $ 14,100/month
route 70% to a small model $ 4,724/month
semantic cache 20% hit rate $ 3,779/month
total saving: 91%The order matters: prompt caching and trimming are low-risk, so do them first; routing and semantic caching change answers, so they need evals and monitoring. Real savings depend on your traffic.
A real example
Example
A writing assistant goes from loss-making to profitable
A writing app charged $12 per user per month but spent about $15 per active user on model calls. Measuring by feature showed "rewrite paragraph" was 60% of spend: it resent a 6,000-token style guide every time and always used the largest model.
The team cached the style guide (cached at a tenth of the price), cut history to the current document section, set max_tokens to 1.5× the input paragraph's length, and routed short rewrites to a small model after evals showed no quality drop. Spend fell to about $3.20 per user; user ratings didn't change.
The easy way
Start here
Turn on prompt caching with a stable prompt prefix, set max_tokens on every call, and log cost per request by feature. Those three steps alone usually cut a large share of the bill and show you where the rest goes.
When you grow: the production setup
Teams run cost as a product metric: per-feature and per-tenant dashboards, budgets with alerts, a gateway that routes by difficulty and falls back on errors, caching layers, batch pipelines for offline work, regular reviews of model choices as prices change, and an eval gate that blocks any cost change that hurts quality.
Trade-offs
What you gain
- Large savings are common (often well over half).
- Faster responses usually come along with smaller prompts and models.
- Better unit economics make features viable.
What it costs you
- Routing, caching and trimming can lower quality if not evaluated.
- More moving parts (routers, caches, batch pipelines).
- Prices change, so the best setup changes too.
When not to use it
- Don't optimise a prototype nobody uses yet; get quality right first, then cut cost.
Common mistakes
- Optimising without measuring which feature costs the most.
- Changing models to save money without running evals.
- Breaking prompt caching by putting a timestamp or user name at the start of the prompt.
- No
max_tokens, so long outputs (and reasoning) drive cost. - Counting cost per call instead of per successful task.
Interview questions
Try answering out loud first, then open the model answer.
Your LLM bill doubled last month. How do you investigate and fix it?
Model answer
Break spend down by feature, model, tenant and token type (input, cached, output) to find what grew: traffic, prompt length, retrieval size, agent steps, retries or a model change. Then apply the matching levers: prompt caching, trimming context, capping output, routing easy traffic to smaller models, caching repeated answers, batch APIs for offline work and step limits for agents, verifying each change with evals and adding cost alerts.
How do you reduce cost without hurting quality?
Model answer
Start with changes that don't alter outputs (prompt caching, removing unused context, batch for offline jobs), then changes that might (trimming, smaller models, routing, semantic caching), each validated against an eval set and monitored online. Measure cost per successful task so cheaper-but-worse setups don't look like wins.
When does self-hosting an open model save money?
Model answer
When volume is high and steady enough to keep GPUs busy, the open model meets the quality bar for the task, and the team can run serving infrastructure. Compare API cost per million tokens with GPU cost divided by real throughput at your utilisation, plus engineering and on-call cost. At low or spiky volume, APIs are usually cheaper.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Prompt caching
- A provider feature that reuses the processed beginning of a prompt when it repeats exactly, making those tokens much cheaper and faster.
- Model routing
- Sending each request to the most suitable model, typically cheap models for easy requests and strong models for hard ones.
- Semantic cache
- A cache that returns a stored answer when a new question is close in meaning (by embedding similarity) to one already answered, skipping the model call.
- Batch API
- A way to submit many requests at once for processing within hours, at a large discount (often about 50%).
- Token budget
- A limit on how many tokens a request, step or whole task may use, to control cost, latency and context size.
- Unit economics
- The cost and revenue of one unit of the product, such as one request, one task or one user per month. AI features must cost less to serve than they earn.
Before and after
Progress is saved in this browser only.