Area 07 // DeploymentLesson 119 / 14014 min read

Autoscaling and load balancing

Autoscaling adds or removes copies of your service as demand changes, and a load balancer spreads incoming requests across those copies, so an AI app stays fast at peak without paying for idle capacity, as long as you scale on the right signal and balance by real load.

On this page

Think of it like this
The problem it solves
How it works: autoscaling
How it works: load balancing
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A supermarket opens more checkouts when queues grow and closes them when it's quiet (autoscaling). A staff member at the front points each shopper to the shortest queue, not just the next till in order, because someone with a full trolley takes far longer than someone with a sandwich (load balancing by real load).

The problem it solves

AI traffic is spiky: a launch, a newsletter or Monday morning can multiply load in minutes. With a fixed number of servers you either pay for idle capacity all night or fall over at peak. And LLM requests are wildly uneven: one asks for 20 tokens, another for 4,000. Spreading them naively leaves some servers swamped while others sit idle.

How it works: autoscaling

Make the service stateless so any replica can handle any request (state in databases and caches).
Pick the signal:
- API servers that call hosted models: CPU or requests per second per replica.
- Self-hosted LLM servers: queue length, concurrent requests, GPU KV-cache use, or time to first token (vLLM exposes metrics like these).
- Workers: queue depth and job age.
Set the rule: Kubernetes' Horizontal Pod Autoscaler computes desired = ceil(current × metric / target) within min and max replicas (see the code and k8s/hpa.yaml in the capstone).
Set bounds: a minimum for latency and redundancy (at least 2), a maximum for cost and provider rate limits (rate limits).
Stabilise: scale up quickly, scale down slowly (stabilisation windows) to avoid flapping.
Account for start-up time: model servers can take minutes to load weights; scale earlier, keep warm capacity, cache weights locally.
Scale nodes too: a cluster autoscaler or managed platform adds machines (including GPU nodes) when pods can't be placed.

How it works: load balancing

A load balancer gives one address for many replicas, terminates HTTPS, and forwards each request to a healthy replica.
Health checks: only replicas passing readiness get traffic (health checks).
Algorithms:
- Round robin: next replica in turn; fine when requests cost about the same.
- Least connections / least outstanding requests: send to the least busy; better for uneven LLM calls (see the code).
- Consistent hashing / session affinity: send related requests to the same replica (e.g. to reuse a prefix cache).
Long-lived connections: streaming responses (SSE) need generous timeouts and connection draining on deploys.
Layers: a global load balancer across regions, then per-region balancers or Kubernetes Services, then sometimes an LLM-aware router in front of model servers.

The load balancer spreads requests over healthy replicas; the autoscaler changes how many there are.

See it in code

Needs: Nothing: plain Python 3. Simplified: the real HPA also waits about 5 minutes before scaling down.

The HPA formula through a traffic spike, then round robin vs least-outstanding-requests on uneven LLM requests.

scaling.py

import itertools, math
 
# AUTOSCALING: the Kubernetes HPA rule: desired = ceil(current * currentMetric / targetMetric), within min/max.
def desired_replicas(current: int, metric: float, target: float, lo=2, hi=10) -> int:
    return max(lo, min(hi, math.ceil(current * metric / target)))
 
replicas = 2
for minute, cpu in enumerate([35, 58, 95, 120, 110, 70, 40, 20]):
    replicas = desired_replicas(replicas, cpu, target=60)
    print(f"min {minute}: avg CPU {cpu:3}% -> {replicas} replicas")
 
# LOAD BALANCING: round robin vs least-outstanding-requests with uneven LLM request lengths.
durations = [9, 1, 1, 1, 8, 1, 1, 1, 7, 1, 1, 1]          # seconds per request (long generations!)
def simulate(pick):
    busy_until = [0.0, 0.0, 0.0]
    finish = []
    for t, d in enumerate(durations):                      # one request arrives per second
        s = pick(busy_until, t)
        start = max(t, busy_until[s])
        busy_until[s] = start + d
        finish.append(start + d - t)                       # time the user waited
    return sum(finish) / len(finish)
rr = itertools.cycle(range(3))
print(f"round robin       : avg {simulate(lambda b, t: next(rr)):.1f}s per request")
print(f"least outstanding : avg {simulate(lambda b, t: min(range(3), key=lambda s: max(b[s], t))):.1f}s per request")

Output

min 0: avg CPU  35% -> 2 replicas
min 1: avg CPU  58% -> 2 replicas
min 2: avg CPU  95% -> 4 replicas
min 3: avg CPU 120% -> 8 replicas
min 4: avg CPU 110% -> 10 replicas
min 5: avg CPU  70% -> 10 replicas
min 6: avg CPU  40% -> 7 replicas
min 7: avg CPU  20% -> 3 replicas
round robin       : avg 4.8s per request
least outstanding : avg 2.8s per request

Round robin kept sending every third request to the replica stuck on a 9-second generation; least-outstanding routed around it and cut the average wait by 40%.

A real example

Example

Scaling on the wrong metric

A team self-hosted a model on GPU nodes and set the autoscaler to CPU utilisation like their other services. During a traffic spike, CPU stayed at 25% (the work was on the GPU), the autoscaler did nothing, and time to first token climbed to 20 seconds as requests queued inside the model server.

They switched to scaling on the server's number of waiting requests, kept a minimum of two warm replicas, pre-cached weights on node disks so new replicas were ready in 90 seconds instead of 6 minutes, and changed the load balancer to least-outstanding-requests. The next spike scaled from 2 to 7 replicas with p95 TTFT under 2 seconds.

The easy way

Start here

On a managed platform, set min instances to 1–2, max to what your budget and provider limits allow, and the concurrency per instance; the platform handles balancing. On Kubernetes, start with an HPA on CPU for API services and a Service in front.

When you grow: the production setup

Teams scale on custom metrics (queue depth, in-flight requests, KV-cache utilisation) via metrics adapters or event-driven autoscalers, tune stabilisation windows, pre-warm capacity before known peaks, run cluster autoscaling for GPU node pools, use LLM-aware routing (least load, prefix-cache affinity), and load-test the whole path regularly.

Trade-offs

What you gain

Handles spikes without paying for peak capacity all day.
Load balancing improves latency and resilience.
Unhealthy replicas are taken out automatically.

What it costs you

Scaling lags behind sudden spikes, especially with slow-starting model servers.
Wrong metrics lead to no scaling or flapping.
More moving parts to tune and observe.

When not to use it

Very low, steady traffic: one or two fixed replicas are simpler.

Common mistakes

Scaling GPU model servers on CPU.
A minimum of one replica, so every deploy or crash causes downtime.
No max, so a traffic flood (or attack) scales costs without limit.
Round robin for highly variable LLM requests.
Ignoring model load time when planning scale-up.

Interview questions

Try answering out loud first, then open the model answer.

How would you autoscale an LLM inference service?

Model answer

Scale horizontally on signals that reflect LLM load (queue length, concurrent requests, KV-cache utilisation or TTFT) rather than CPU; set a minimum of warm replicas and a maximum bound by budget and GPU quota; scale up fast and down slowly; reduce start-up time by caching weights; add GPU node autoscaling; and load-test to validate.

Which load-balancing algorithm suits LLM APIs and why?

Model answer

Least outstanding requests (or least connections), because request durations vary enormously with prompt and output length; round robin would keep sending work to replicas busy with long generations. Add health-based routing, and optionally prefix-cache affinity to reuse cached prompts.

What is the HPA formula and what are its pitfalls?

Model answer

desired = ceil(currentReplicas × currentMetric / targetMetric), clamped to min and max. Pitfalls: choosing a metric that doesn't reflect the bottleneck, flapping without stabilisation, slow pod start-up making scaling lag, and missing resource requests (CPU utilisation is measured against requests).

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Autoscaling: Automatically adding or removing copies of a service (or machines) as demand changes, within minimum and maximum limits.
Load balancer: A component that receives requests at one address and spreads them across healthy replicas.
Replica: One running copy of a service. Several replicas behind a load balancer share traffic and survive individual failures.
Horizontal Pod Autoscaler (HPA): The Kubernetes component that changes the number of pod replicas based on a metric such as CPU or queue length.
Least connections: A load-balancing method that sends each new request to the replica with the fewest requests in progress; good when request lengths vary a lot.
Scale to zero: Running no instances when there is no traffic, so you pay nothing while idle, at the cost of a cold start on the next request.

Before and after

Read first