Area 05 of 08

AI System Design

Running LLM apps at scale: GPUs, serving, batching, caching, routing, rate limits, cost, and full interview-style designs.

Printable cheat sheet for this area

1. 078

   GPUs, memory and GPU vs API cost

   Models run on GPUs because generating text is mostly huge matrix multiplications and moving weights through memory, so what limits a deployment is usually GPU memory (VRAM) for the weights plus the KV cache, and renting GPUs only beats paying an API per token when the GPUs stay busy.

   CoversMemory and GPU basics (VRAM, why models need GPUs) · GPU vs API Cost

2. 079

   Streaming, time to first token and throughput

   Streaming sends the answer to the user token by token as it's generated, so they start reading after the time to first token instead of waiting for the whole reply, and every serving system trades per-user speed (latency) against how many users it can serve at once (throughput).

   CoversStreaming Responses · Time to First Token · Throughput vs Latency

3. 080

   Serving models with vLLM

   vLLM is a popular open-source inference server that runs open models fast and cheaply, mainly thanks to continuous batching (new requests join the batch at every step) and PagedAttention (the KV cache is stored in small pages, like computer memory), and it exposes an OpenAI-compatible API.

   CoversInference Serving with vLLM · Continuous Batching · PagedAttention

4. 081

   Speculative decoding and prefix caching

   Speculative decoding speeds up generation by letting a small fast model guess several tokens ahead and having the big model check them all in one pass, and prefix caching skips recomputing the start of a prompt that many requests share, cutting time to first token and cost.

   CoversSpeculative Decoding · Prefix Caching

5. 082

   Model routing, fallbacks and LLM gateways

   Model routing sends each request to the cheapest model that can handle it, fallbacks automatically retry on another model or provider when one fails, and an LLM gateway puts both (plus keys, limits, caching and logging) in one layer between your apps and every model.

   CoversModel Routing · Model Fallbacks · LLM gateways (e.g. LiteLLM)

6. 083

   Semantic caching

   A semantic cache stores past answers and returns one when a new question means the same thing as a cached one (judged by embedding similarity), skipping the model call entirely, which saves cost and time as long as the similarity threshold is strict enough to avoid wrong answers.

7. 084

   Token budgets and context window management

   A token budget is a planned limit on how many input and output tokens each feature, request or task may use, and context window management is the system that decides what to keep, trim, summarise or retrieve so every request fits its budget and the model's window.

   CoversToken Budgets · Context Window Management

8. 085

   Batch inference and async job pipelines

   Work that doesn't need an instant answer (summarising a million documents, nightly classification, long agent tasks) should run as background jobs, either through a provider's batch API at a large discount or through your own queue of workers, instead of inside a user's web request.

   CoversBatch Inference · Async Job Pipelines

9. 086

   Provider rate limits and multi-tenant design

   Model providers cap how many requests and tokens you can send per minute, so your system must spread that shared capacity across all your customers fairly (multi-tenancy), with per-tenant quotas, queues and isolation so one big customer can't slow down or see data from the others.

   CoversProvider Rate Limits · Multi-tenant Design

10. 087

    RAG vs fine-tuning

    RAG gives a model the right knowledge at question time (fresh, citable, permission-aware), while fine-tuning changes how the model behaves (style, format, narrow skills), so most systems use RAG for knowledge, fine-tuning only for behaviour, and often both together.

11. 088

    Where guardrails sit in the architecture

    Guardrails are checks placed around the model at several points (on the way in, around retrieval and tools, and on the way out) so that unsafe input is stopped, risky actions need permission, and bad output never reaches the user, with each check chosen to fit the latency budget.

12. 089

    Cost optimization

    LLM cost is roughly requests × tokens × price per token, so you cut it by sending fewer tokens (trimming, caching), paying less per token (smaller models, routing, batch discounts) and making fewer calls (semantic caching, fewer agent steps), always checking with evals that quality holds.

13. 090

    Case study: a chatbot for 1 million users

    Designing a customer chatbot for a million daily users means estimating traffic and tokens, choosing models and a gateway, streaming answers, storing conversations, grounding answers with retrieval, and adding caching, guardrails, rate limits and monitoring, then working out what it will cost.

14. 091

    Case study: a document Q&A system

    Designing a document Q&A system for a large company means building an ingestion pipeline (parse, chunk, embed, index, keep in sync), a permission-aware retrieval and answering path with citations, and evals and monitoring, sized from the number of documents, queries and tokens.

15. 092

    Case study: an AI coding assistant

    An AI coding assistant has two very different workloads, instant autocomplete with a small fast model and short context, and chat or agent tasks with a strong model, repository context and tools that run in a sandbox, so the design separates them and controls latency, context, safety and cost for each.