Skip to content
← All cheat sheets

Cheat sheet · Area 05

AI System Design

GPUs, memory and GPU vs API cost

Models run on GPUs because generating text is mostly huge matrix multiplications and moving weights through memory, so what limits a deployment is usually GPU memory (VRAM) for the weights plus the KV cache, and renting GPUs only beats paying an API per token when the GPUs stay busy.

  • VRAM needed ≈ weights (params × bits ÷ 8) + KV cache (grows with context × concurrent users) + ~10–20% overhead.
  • 8B model: ~16 GB at 16-bit, ~5 GB at 4-bit, before KV cache. 70B: ~140 GB at 16-bit.
  • Decode speed is limited by memory bandwidth (reading weights for every token), not just raw FLOPs.
  • Common data-centre GPUs (as of Nov 2026): 24 GB (L4/A10 class), 80 GB (H100), 141 GB (H200), 180–192 GB (B200 class).
  • GPU cost per token = hourly price ÷ tokens produced per hour. Idle GPUs make self-hosting expensive.

Streaming, time to first token and throughput

Streaming sends the answer to the user token by token as it's generated, so they start reading after the time to first token instead of waiting for the whole reply, and every serving system trades per-user speed (latency) against how many users it can serve at once (throughput).

  • Total latency ≈ TTFT (queue + prefill) + output tokens × time per token.
  • Streaming doesn't make generation faster; it makes waiting feel shorter (users read as it arrives).
  • TTFT grows with prompt length (prefill) and queueing; time per token grows with model size and batch load.
  • Throughput = tokens/s (or requests/s) the system produces in total. Bigger batches ↑ throughput, ↓ per-user speed.
  • Stream with SSE (most LLM APIs) or WebSockets; handle disconnects and partial output.

Serving models with vLLM

vLLM is a popular open-source inference server that runs open models fast and cheaply, mainly thanks to continuous batching (new requests join the batch at every step) and PagedAttention (the KV cache is stored in small pages, like computer memory), and it exposes an OpenAI-compatible API.

  • vllm serve <model> → an OpenAI-compatible API (default port 8000); clients just change base_url.
  • Continuous batching: finished requests leave and waiting ones join at every decode step → high GPU use, low queueing.
  • PagedAttention: KV cache in fixed-size blocks allocated on demand → little wasted memory, more concurrent requests, prefix sharing.
  • Also: prefix caching, speculative decoding, quantization (FP8, AWQ/GPTQ, INT4), tensor/pipeline/expert parallelism, multi-LoRA.
  • Alternatives: SGLang, TensorRT-LLM / NVIDIA NIM, Hugging Face TGI, llama.cpp/Ollama for local.

Speculative decoding and prefix caching

Speculative decoding speeds up generation by letting a small fast model guess several tokens ahead and having the big model check them all in one pass, and prefix caching skips recomputing the start of a prompt that many requests share, cutting time to first token and cost.

  • Speculative decoding: draft k tokens cheaply → big model verifies all k in one forward pass → keep the accepted ones.
  • Output is the same as the big model alone (for exact verification); speed-up depends on the acceptance rate.
  • Draft sources: a small model, n-gram/prompt lookup, or extra prediction heads (EAGLE-style).
  • Prefix caching: store the KV cache of shared prompt beginnings (system prompt, documents) and reuse it.
  • Provider prompt caching is the hosted version of the same idea.

Model routing, fallbacks and LLM gateways

Model routing sends each request to the cheapest model that can handle it, fallbacks automatically retry on another model or provider when one fails, and an LLM gateway puts both (plus keys, limits, caching and logging) in one layer between your apps and every model.

  • Routing: easy requests → small/cheap model; hard ones → strong model. Decide by rules, a classifier, or a cheap model.
  • Fallbacks: on errors, timeouts, 429s or overloads, retry on a backup model/provider (with a timeout budget).
  • Circuit breaker: stop sending traffic to a failing model for a while instead of retrying it every time.
  • LLM gateway (e.g. LiteLLM proxy, cloud AI gateways): one API in front of all providers with routing, fallbacks, keys, budgets, rate limits, caching, logging.
  • Evaluate each route: a fallback model must pass your evals too.

Semantic caching

A semantic cache stores past answers and returns one when a new question means the same thing as a cached one (judged by embedding similarity), skipping the model call entirely, which saves cost and time as long as the similarity threshold is strict enough to avoid wrong answers.

  • Exact cache: same text → same answer. Semantic cache: similar meaning → reuse the answer.
  • Embed the question, find the nearest cached question, return its answer if similarity ≥ threshold.
  • Threshold too low → wrong answers ('reset router' gets the 'reset password' answer). Too high → few hits.
  • Never share cached answers across users when answers depend on the user, their data or their permissions.
  • Expire entries (TTL) and invalidate when the underlying data changes.

Token budgets and context window management

A token budget is a planned limit on how many input and output tokens each feature, request or task may use, and context window management is the system that decides what to keep, trim, summarise or retrieve so every request fits its budget and the model's window.

  • Budget per feature: max input tokens, max output tokens (max_tokens), max total per task or per user per day.
  • Fill the window by priority: system + current question → essential state → retrieved docs → recent history → older history (summarised).
  • Trim strategies: sliding window, summarise old turns, drop stale tool results, retrieve instead of including.
  • Count tokens with the provider's counter before sending; leave room for the output.
  • Budgets protect cost, latency and quality (less noise), and stop runaway prompts.

Batch inference and async job pipelines

Work that doesn't need an instant answer (summarising a million documents, nightly classification, long agent tasks) should run as background jobs, either through a provider's batch API at a large discount or through your own queue of workers, instead of inside a user's web request.

  • Batch API: submit many requests in one file/job, results within hours, ~50% cheaper (OpenAI, Anthropic, Google).
  • Async job pipeline: API returns a job ID immediately → queue → workers call the model → store result → notify/poll.
  • Limit concurrency to stay under provider rate limits; retry with backoff; dead-letter jobs that keep failing.
  • Make jobs idempotent so retries don't duplicate work or side effects.
  • Use for: bulk processing, long agent tasks, anything over a few seconds that a user doesn't watch live.

Provider rate limits and multi-tenant design

Model providers cap how many requests and tokens you can send per minute, so your system must spread that shared capacity across all your customers fairly (multi-tenancy), with per-tenant quotas, queues and isolation so one big customer can't slow down or see data from the others.

  • Provider limits: requests/min (RPM) and tokens/min (input and output TPM), by model and account tier; 429 = slow down (respect retry-after).
  • Your app shares ONE provider quota between ALL tenants → allocate it with per-tenant quotas (token buckets).
  • Noisy neighbour: one tenant's burst blocks others. Fix: per-tenant limits, fair queues, priority tiers.
  • Isolation: data (retrieval filters, caches, memory scoped by tenant), cost tracking and keys per tenant.
  • Scale limits: higher provider tiers, multiple keys/regions/providers via a gateway, batch for bulk.

RAG vs fine-tuning

RAG gives a model the right knowledge at question time (fresh, citable, permission-aware), while fine-tuning changes how the model behaves (style, format, narrow skills), so most systems use RAG for knowledge, fine-tuning only for behaviour, and often both together.

  • RAG = knowledge: facts that change, private documents, citations, per-user permissions.
  • Fine-tuning = behaviour: consistent format and tone, narrow tasks, shorter prompts, smaller/cheaper models.
  • Try in order: prompting → RAG (if knowledge is missing) → fine-tuning (if behaviour still isn't right) → both.
  • Fine-tuning facts in is unreliable and they go stale; you can't cite or permission-filter weights.
  • Decide with evals: compare prompted, RAG, fine-tuned and combined variants on the same test set.

Where guardrails sit in the architecture

Guardrails are checks placed around the model at several points (on the way in, around retrieval and tools, and on the way out) so that unsafe input is stopped, risky actions need permission, and bad output never reaches the user, with each check chosen to fit the latency budget.

  • Layers: input guard → context/retrieval filters → model → tool/action guard → output guard → logging.
  • Input: injection and abuse detection, PII redaction, topic limits, size limits.
  • Output: PII and secret masking, policy and safety checks, schema validation, grounding checks.
  • Actions: allow-lists, least privilege, human approval for risky steps; the model never holds the keys.
  • Latency: run cheap guards in parallel with the model; reserve slow guards for risky paths.

Cost optimization

LLM cost is roughly requests × tokens × price per token, so you cut it by sending fewer tokens (trimming, caching), paying less per token (smaller models, routing, batch discounts) and making fewer calls (semantic caching, fewer agent steps), always checking with evals that quality holds.

  • Cost ≈ requests × (input tokens × input price + output tokens × output price).
  • Fewer tokens: trim history and retrieval, shorter prompts, cap max_tokens, prompt caching for stable prefixes.
  • Cheaper tokens: route easy requests to small models, batch API (~50% off), self-host at high steady volume.
  • Fewer calls: semantic/exact caching, fewer agent steps, avoid needless retries.
  • Measure cost per request and per user first; every change needs an eval check.

Case study: a chatbot for 1 million users

Designing a customer chatbot for a million daily users means estimating traffic and tokens, choosing models and a gateway, streaming answers, storing conversations, grounding answers with retrieval, and adding caching, guardrails, rate limits and monitoring, then working out what it will cost.

  • Start with requirements and numbers: DAU, messages per user, tokens per message, peak factor.
  • 1M DAU × 6 messages ≈ 69 average QPS, ~208 at peak; ~62,500 output tokens/s at peak.
  • Architecture: client → API (auth, rate limits) → chat service (context assembly, RAG) → gateway (routing, fallbacks, cache) → model; stream with SSE.
  • Store conversations in a database; keep recent turns + a summary in the prompt.
  • Cost is the big decision: model choice and caching change the bill by 20×.

Case study: a document Q&A system

Designing a document Q&A system for a large company means building an ingestion pipeline (parse, chunk, embed, index, keep in sync), a permission-aware retrieval and answering path with citations, and evals and monitoring, sized from the number of documents, queries and tokens.

  • Two paths: ingestion (async, batch) and query (online, low latency).
  • Ingestion: connectors → parse → chunk → embed → index with metadata (source, ACL, date); re-sync on change.
  • Query: auth → rewrite → hybrid search filtered by permissions → rerank → answer with citations → guard.
  • Sizing: 200K docs ≈ 1.5B tokens ≈ 3.3M chunks ≈ 20 GB of float32 vectors; embedding costs about $30.
  • Query cost dominates; evals on retrieval and answers decide quality.

Case study: an AI coding assistant

An AI coding assistant has two very different workloads, instant autocomplete with a small fast model and short context, and chat or agent tasks with a strong model, repository context and tools that run in a sandbox, so the design separates them and controls latency, context, safety and cost for each.

  • Two workloads: autocomplete (hundreds of ms, small model, tiny output) and chat/agent (seconds to minutes, strong model, tools).
  • Context is the product: current file, cursor, open tabs, related symbols, repo search, project rules.
  • Agents edit files and run tests in a sandbox; risky commands need approval; never ship secrets in context.
  • 5,000 devs: ~3.6M completions/day, ~333 QPS peak; agents cost far more than autocomplete.
  • Measure: acceptance rate, latency p95, agent task success on a benchmark of real tasks, cost per developer.