Area 07 // DeploymentLesson 117 / 14014 min read

GPU instances

A GPU instance is a rented cloud machine with one or more GPUs, used to run or fine-tune open models yourself; choosing one means checking that the model fits in GPU memory, estimating throughput and cost per token, and comparing that with simply calling a model API.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Renting a van to move house: first check the furniture fits (memory), then how many trips you'll need (throughput), then compare the daily rental with paying a removal company per box (the API). Renting only makes sense if the van will be full and busy; a van parked half-empty for a week is money down the drain.

The problem it solves

You want to run an open model yourself, for privacy, cost at high volume, customisation, or offline needs (open vs closed models). Cloud providers offer dozens of GPU types at very different prices. Pick too small and the model won't load; too big and you burn money; leave it idle and an API would have been far cheaper.

How it works, step by step

Estimate memory: weights = parameters × bytes per parameter (FP16 2, INT8 1, INT4 0.5), plus ~20% overhead, plus KV cache for your context lengths and concurrent users (GPUs and memory, KV cache).
Consider quantization to fit a smaller GPU or leave room for more concurrent requests, and check quality on your evals (quantization).
Pick the instance: a GPU with enough VRAM (or several GPUs with tensor parallelism for big models), plus enough CPU RAM and fast disk for loading weights.
Measure throughput, not guess: run a serving engine (vLLM) with realistic prompts and concurrency, and record tokens per second and latency (TTFT, time per token).
Compute cost per million tokens = hourly price ÷ (tokens per second × 3,600) × 1,000,000, at your real utilisation; compare with API prices (cost optimization).
Choose a pricing model: on-demand (flexible), reserved or committed (cheaper for steady use), spot (much cheaper, can be reclaimed: good for batch jobs with checkpoints), or serverless GPU platforms that bill per second and scale to zero.
Operate it: GPU drivers and CUDA versions, model weights cached on fast storage, health checks, autoscaling on queue depth or GPU utilisation (autoscaling), monitoring of GPU memory and utilisation.
Check availability and quotas: popular GPUs can be scarce in some regions; request quota early.

See it in code

Needs: Nothing: plain Python 3. Instance names, prices and throughputs are illustrative examples, not quotes; measure your own.

Which instance can hold which model, and what a busy GPU costs per million tokens.

gpu_sizing.py

# Which GPU instance can serve this model? Memory first, then cost per million tokens.
def vram_needed_gb(params_b: float, bytes_per_param: float, kv_cache_gb: float, overhead=1.2) -> float:
    return params_b * bytes_per_param * overhead + kv_cache_gb
 
# Example instance types and hourly prices (illustrative on-demand prices, November 2026; check your cloud).
INSTANCES = [("1x L4 24GB", 24, 0.80), ("1x L40S 48GB", 48, 1.90), ("1x H100 80GB", 80, 4.50)]
 
for model, params_b in [("8B model", 8), ("32B model", 32)]:
    for precision, bpp in [("FP16", 2), ("INT4", 0.5)]:
        need = vram_needed_gb(params_b, bpp, kv_cache_gb=6)
        fits = [name for name, vram, _ in INSTANCES if vram >= need]
        print(f"{model} {precision}: needs ~{need:.0f} GB -> {fits[0] if fits else 'needs multiple GPUs'}")
 
# Cost per 1M output tokens if the server sustains a given throughput (batching many users).
for name, vram, price in INSTANCES:
    tokens_per_s = {"1x L4 24GB": 600, "1x L40S 48GB": 1_500, "1x H100 80GB": 4_000}[name]
    per_million = price / (tokens_per_s * 3600) * 1e6
    print(f"{name}: ${price:.2f}/h at {tokens_per_s:,} tok/s -> ${per_million:.2f} per 1M tokens (if kept busy)")

Output

8B model FP16: needs ~25 GB -> 1x L40S 48GB
8B model INT4: needs ~11 GB -> 1x L4 24GB
32B model FP16: needs ~83 GB -> needs multiple GPUs
32B model INT4: needs ~25 GB -> 1x L40S 48GB
1x L4 24GB: $0.80/h at 600 tok/s -> $0.37 per 1M tokens (if kept busy)
1x L40S 48GB: $1.90/h at 1,500 tok/s -> $0.35 per 1M tokens (if kept busy)
1x H100 80GB: $4.50/h at 4,000 tok/s -> $0.31 per 1M tokens (if kept busy)

"If kept busy" is the catch: at 10% utilisation, each cost per token is ten times higher. Compare with small hosted models at about $0.50 per million output tokens (as of November 2026) and self-hosting only wins at steady, high volume or for privacy reasons.

A real example

Example

Self-hosting for a data-residency requirement

A healthcare company couldn't send patient notes to external APIs. They chose an open 8B model, quantized to 8-bit after evals showed no meaningful quality loss on their summarisation set, served with vLLM on two single-GPU instances in their own cloud region behind a load balancer. Measured throughput was about 1,200 tokens per second per GPU at their typical prompt length; at their daytime traffic the GPUs ran at 60–70% utilisation, and a scheduled scale-down to one instance overnight cut the bill by a third. Weights were cached on local disk so a replacement instance became ready in under three minutes.

The easy way

Start here

Before renting anything, estimate memory with the formula above and run the model on a serverless GPU platform or a single on-demand instance with vLLM for a day of realistic tests. Compare cost per million tokens at your real traffic with the API price.

When you grow: the production setup

Teams benchmark several GPU types with their own prompts, use quantized variants validated by evals, mix reserved capacity for the baseline with on-demand or spot for peaks and batch jobs, autoscale on queue depth, pre-cache weights, monitor GPU utilisation and memory, and revisit choices as new GPUs and prices arrive.

Trade-offs

What you gain

Full control over model, data and location.
Cheapest per token at high, steady utilisation.
No provider rate limits.

What it costs you

You pay for idle time.
Operations work: drivers, serving, scaling, upgrades.
GPU availability and quotas can be limited.

When not to use it

Low or spiky traffic with no privacy requirement: use an API or a serverless GPU platform.

Common mistakes

Sizing only for the weights and forgetting the KV cache.
Comparing cost at 100% utilisation when real traffic is far lower.
Using spot instances for serving without fallback capacity.
Skipping evals after quantizing.
Downloading weights from the internet on every start.

Interview questions

Try answering out loud first, then open the model answer.

How do you choose a GPU instance to serve an open model?

Model answer

Estimate memory for weights at the chosen precision plus overhead and KV cache for target context and concurrency; pick GPUs that fit (or plan tensor parallelism); benchmark throughput and latency with a serving engine on realistic load; compute cost per million tokens at expected utilisation and compare with APIs; choose pricing (reserved, on-demand, spot) by workload; and plan operations (autoscaling, monitoring, weight caching).

How would you estimate the cost per token of self-hosting?

Model answer

Measure sustained tokens per second at realistic batch sizes and prompt lengths, multiply by 3,600 for tokens per hour, adjust by expected utilisation, then divide the hourly instance price by that and scale to per million tokens; add costs for redundancy, storage, networking and engineering time.

When are spot instances a good idea for AI workloads?

Model answer

For interruptible work: batch inference, evaluations, data processing and training with frequent checkpoints. For online serving they're risky unless combined with on-demand capacity and graceful draining, because instances can be reclaimed at short notice.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

GPU (graphics processing unit): A chip with thousands of small cores that do many multiplications at once, which is why it runs neural networks far faster than a CPU. Its memory (VRAM) decides which models fit.
VRAM: The memory on a GPU. Model weights, the KV cache and activations must fit in it, so it usually limits which models a GPU can run.
GPU instance: A cloud virtual machine with one or more GPUs attached, rented by the hour to run or fine-tune models.
Quantization: Storing a model's numbers with fewer bits (e.g. 4-bit instead of 16-bit) to save memory and speed it up, with a small quality loss.
Throughput: How much work a system finishes per second, e.g. requests per second or tokens per second across all users. Batching raises throughput, sometimes at the cost of per-request latency.
Spot instance: Spare cloud capacity sold at a large discount that the provider can take back at short notice; good for interruptible jobs.

Before and after

Read first