Area 05 // AI System DesignLesson 78 / 14015 min read
GPUs, memory and GPU vs API cost
Models run on GPUs because generating text is mostly huge matrix multiplications and moving weights through memory, so what limits a deployment is usually GPU memory (VRAM) for the weights plus the KV cache, and renting GPUs only beats paying an API per token when the GPUs stay busy.
Think of it like this
Analogy
A GPU is a kitchen with thousands of cooks working at once but a small fridge (VRAM). The cooks are amazingly fast at chopping in parallel (matrix maths), but every ingredient (model weight) has to fit in the fridge, along with a tray of notes for each customer currently being served (the KV cache). Too big a recipe or too many customers, and it doesn't fit.
Renting that kitchen costs the same per hour whether it's full of orders or empty, while ordering takeaway (an API) costs per dish.
The problem it solves
When teams move from calling APIs to running models themselves, the first questions are: which GPU, how many, and is it even cheaper? People size by "it's an 8B model, any GPU will do" and then hit out-of-memory errors at 20 concurrent users, or rent expensive GPUs that sit idle 80% of the time and cost far more per token than an API. You need back-of-the-envelope maths for memory and cost.
How it works: why models need GPUs
- Inference is matrix maths: every generated token multiplies vectors by the model's weight matrices (transformer). A GPU has thousands of cores that do these multiply-adds in parallel.
- Weights must sit in GPU memory (VRAM). An 8B-parameter model at 16-bit needs about 16 GB; at 4-bit about 4–5 GB (quantization).
- The KV cache needs room too: for every active request, keys and values for every token in its context (KV cache). It grows with context length × concurrent users and often rivals the weights.
- Plus overhead: activations, buffers and the serving engine, roughly 10–20%.
- Decode is memory-bound: each new token reads most of the weights from VRAM, so memory bandwidth (TB/s) limits tokens per second per request; batching many requests shares each read across users.
- Big models span several GPUs: tensor or pipeline parallelism splits a model across GPUs connected by fast links (NVLink).
- GPU classes (as of November 2026): 24 GB inference cards (L4 / A10 class) for small models; 80 GB H100 and 141 GB H200 for larger ones; 180–192 GB B200-class for the biggest. Cloud prices vary widely by provider, region and commitment.
How it works: GPU vs API cost
- API cost = tokens × price per token, with no idle cost (token pricing).
- GPU cost per token = hourly price ÷ tokens produced per hour. Tokens per hour depends on the model, batching and the serving engine (vLLM).
- Utilisation decides everything: at 100% busy a GPU can be cheaper per token than a small API model; at 10% busy it's many times more expensive.
- Add hidden costs: engineers, on-call, monitoring, spare capacity for peaks, idle nights and weekends, failed nodes.
- Self-hosting usually wins when traffic is high and steady, data must stay in-house, or you need a fine-tuned open model; APIs win for spiky, low or uncertain traffic and for the best closed models.
See it in code
- Needs
- Nothing: plain Python 3. Model shapes and prices are examples (as of November 2026); use your model card and cloud price list.
VRAM needed for 16 users with 8K context each, and when a rented GPU beats an API.
def vram_gb(params_b, bits, ctx_tokens, users, layers, kv_heads, head_dim):
weights = params_b * 1e9 * bits / 8 # the model's numbers
kv_per_token = 2 * layers * kv_heads * head_dim * 2 # keys + values, 16-bit
kv_cache = kv_per_token * ctx_tokens * users # grows with context x users
overhead = 0.10 * (weights + kv_cache) # activations, buffers (rough)
return (weights + kv_cache + overhead) / 1e9
# Example shapes: an 8B model (32 layers, 8 KV heads x 128) and a 70B model (80 layers, 8 KV heads x 128)
for name, p, layers in [("8B", 8, 32), ("70B", 70, 80)]:
for bits in (16, 4):
gb = vram_gb(p, bits, ctx_tokens=8_000, users=16, layers=layers, kv_heads=8, head_dim=128)
print(f"{name} @ {bits:2}-bit, 16 users x 8K context: {gb:6.1f} GB")
# GPU vs API: when does renting a GPU beat paying per token? (example prices, Nov 2026)
gpu_per_hour = 2.50 # one data-centre GPU, on-demand (varies a lot by cloud)
gpu_tokens_per_sec = 2_000 # output tokens/s with good batching (depends on model + load)
api_price_per_m_out = 0.50 # a small API model's output price
gpu_cost_per_m = gpu_per_hour / (gpu_tokens_per_sec * 3600 / 1e6)
print(f"GPU at full load: ${gpu_cost_per_m:.2f} per 1M tokens vs API ${api_price_per_m_out:.2f}")
for utilisation in (1.0, 0.3, 0.1):
print(f" at {utilisation:.0%} busy: ${gpu_cost_per_m / utilisation:.2f} per 1M tokens")8B @ 16-bit, 16 users x 8K context: 36.1 GB
8B @ 4-bit, 16 users x 8K context: 22.9 GB
70B @ 16-bit, 16 users x 8K context: 200.1 GB
70B @ 4-bit, 16 users x 8K context: 84.6 GB
GPU at full load: $0.35 per 1M tokens vs API $0.50
at 100% busy: $0.35 per 1M tokens
at 30% busy: $1.16 per 1M tokens
at 10% busy: $3.47 per 1M tokensNotice the 8B model needs 36 GB for this workload, not 16 GB, because the KV cache for 16 users is bigger than the weights. And the GPU only beats the small API model when it's busy most of the time.
A real example
Example
The internal assistant that cost more on GPUs
A company moved its internal assistant from an API to a self-hosted open model on two rented 80 GB GPUs "to save money". Traffic was 9-to-5 on weekdays: the GPUs sat idle about 75% of the week. Their real cost per token was about four times the API price, before counting the engineer who spent a third of their time keeping it running.
They moved back to an API for the assistant and kept self-hosting only for a nightly batch job that ran the GPUs at full load for six hours (and was much cheaper than the API for that volume).
The easy way
Start here
Before self-hosting, do two calculations: memory (weights + KV cache for your expected users and context + 20%) and cost per million tokens at your realistic utilisation. If the API is cheaper or close, stay on the API.
When you grow: the production setup
Teams benchmark their actual model and traffic with a serving engine, measure tokens per second per GPU at target latency, plan capacity for peaks with autoscaling (GPU instances), use reserved or spot capacity where safe, quantize to fit cheaper GPUs, and track cost per token and utilisation on dashboards.
Trade-offs
What you gain
- GPUs make inference fast enough for real-time use.
- Self-hosting at high utilisation can be much cheaper per token than APIs.
- Full control over models, data and latency.
What it costs you
- VRAM limits model size, context and concurrency.
- Idle GPUs still cost money; capacity planning is hard.
- Operating GPU infrastructure needs specialist skills.
When not to use it
- Low, spiky or unpredictable traffic: pay per token instead.
- You need frontier closed models: they're only available through APIs or cloud platforms.
Common mistakes
- Sizing VRAM from the weights alone and forgetting the KV cache.
- Comparing GPU cost at 100% utilisation with real, much lower usage.
- Ignoring engineering and on-call costs of self-hosting.
- Picking the biggest GPU instead of quantizing to fit a cheaper one.
- Not benchmarking tokens per second on the actual model and workload.
Interview questions
Try answering out loud first, then open the model answer.
How do you estimate GPU memory needed to serve a model?
Model answer
Weights = parameters × bytes per parameter (2 for 16-bit, ~0.5 for 4-bit). KV cache = 2 × layers × KV heads × head dimension × bytes × tokens, summed over concurrent requests' contexts. Add roughly 10–20% for activations and engine overhead. Compare with GPU VRAM; if it doesn't fit, quantize, reduce max context or concurrency, use a smaller model, or shard across GPUs.
When is self-hosting an LLM cheaper than using an API?
Model answer
When sustained utilisation is high enough that hourly GPU cost divided by tokens produced is below the API's per-token price, after including engineering, on-call, redundancy, peak headroom and idle time. That usually means large, steady volumes (often batch workloads) on open models. For spiky or modest traffic, APIs are cheaper and simpler; privacy or customisation can also justify self-hosting regardless of cost.
Why is LLM decoding memory-bandwidth-bound?
Model answer
Each generated token requires reading most of the model's weights (and the KV cache) from GPU memory, while the arithmetic per byte read is small for a single sequence. So the time per token is set by how fast memory can be read. Batching multiple sequences reuses each weight read across many tokens, raising arithmetic intensity and throughput.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- GPU (graphics processing unit)
- A chip with thousands of small cores that do many multiplications at once, which is why it runs neural networks far faster than a CPU. Its memory (VRAM) decides which models fit.
- VRAM
- The memory on a GPU. Model weights, the KV cache and activations must fit in it, so it usually limits which models a GPU can run.
- KV cache
- Stored keys and values of already-processed tokens, so each new token can be generated without recomputing everything before it.
- Quantization
- Storing a model's numbers with fewer bits (e.g. 4-bit instead of 16-bit) to save memory and speed it up, with a small quality loss.
- Memory bandwidth
- How many bytes per second a GPU can read from its memory. Generating each token means reading the model's weights, so bandwidth often limits tokens per second more than raw compute.
- GPU utilisation
- The share of time (or compute) a GPU is doing useful work. Idle or half-busy GPUs are expensive, so serving systems batch requests to keep utilisation high.
Before and after
Progress is saved in this browser only.