Area 07 // DeploymentLesson 118 / 14016 min read

Kubernetes basics

Kubernetes runs containers across a cluster of machines and keeps them in the state you describe, the right number of healthy copies, reachable through a stable address, updated gradually and rolled back when a new version can't get ready, which makes it a common platform for serving AI apps and models at scale.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A hotel manager who is told "keep 50 rooms cleaned and ready at all times". They don't care which cleaner does which room; if a cleaner goes home sick, someone else picks up the work; when the new cleaning method is introduced, they switch a few rooms at a time and keep the old method going until the new one proves itself. Guests only ever see the front desk (the Service), never the staff rota.

The problem it solves

Docker runs containers on one machine. In production you need many machines, several copies of each service, restarts when something crashes, a stable address in front of changing containers, gradual updates, rollbacks, and GPU scheduling for model servers. Doing that by hand across servers is impractical. Kubernetes is the most widely used system for it, offered as a managed service by every major cloud.

How it works, step by step

A cluster has a control plane (the brain) and worker nodes (machines, including GPU nodes) that run your containers.
You declare desired state in YAML and kubectl apply it; controllers keep reality matching it.
Pods are the smallest unit: one or more containers sharing a network address (Docker).
A Deployment says "run 2 replicas of image rag-app:v1 with these settings"; it creates pods, replaces crashed ones and handles updates.
Requests and limits reserve CPU, memory (and GPUs) so the scheduler places pods on nodes with room.
Probes: livenessProbe restarts a stuck container; readinessProbe only sends traffic to pods that are ready (health checks).
A Service gives the pods a stable name and load-balances across ready ones; an Ingress or Gateway exposes it to the internet.
Rolling updates: changing the image or settings creates new pods gradually; with maxUnavailable: 0, old pods keep serving until new ones are ready; progressDeadlineSeconds marks a stuck rollout as failed; kubectl rollout undo goes back.
More pieces: ConfigMaps and Secrets for configuration (secrets), HorizontalPodAutoscaler for scaling (autoscaling), namespaces for separation.

Architecture

A Deployment keeps two ready pods; the Service only sends traffic to pods whose readiness probe passes.

See it in code

Install: kubectl, plus a cluster (kind or minikube locally, or a managed cloud cluster)
Needs: The full manifest is examples/rag-app/k8s/deployment.yaml in the course repository. The run below used a kind cluster inside GitHub Actions.

The capstone's Deployment (key parts), then a real deploy, a bad release that never becomes ready, and a rollback.

k8s/deployment.yaml (key parts)

apiVersion: apps/v1
kind: Deployment
metadata: { name: rag-app }
spec:
  replicas: 2
  progressDeadlineSeconds: 60          # a rollout stuck longer than this is marked failed
  strategy:
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }   # never drop below 2 ready pods
  template:
    spec:
      containers:
        - name: rag-app
          image: rag-app:v1
          resources: { requests: { cpu: 100m, memory: 256Mi }, limits: { memory: 512Mi } }
          livenessProbe: { httpGet: { path: /health, port: 8000 } }
          readinessProbe: { httpGet: { path: /ready, port: 8000 } }
---
apiVersion: v1
kind: Service
metadata: { name: rag-app }
spec:
  selector: { app: rag-app }
  ports: [{ port: 80, targetPort: 8000 }]

Output

$ kubectl apply -f k8s/deployment.yaml && kubectl rollout status deployment/rag-app
deployment.apps/rag-app created
service/rag-app created
Waiting for deployment "rag-app" rollout to finish: 1 of 2 updated replicas are available...
deployment "rag-app" successfully rolled out
$ curl -s localhost:8080/health          # via kubectl port-forward service/rag-app 8080:80
{"status":"ok","chunks":12,"version":"v1"}
 
$ kubectl set env deployment/rag-app LLM_PROVIDER=anthropic    # bad release: no API key, /ready fails
$ kubectl rollout status deployment/rag-app --timeout=75s
Waiting for deployment "rag-app" rollout to finish: 1 out of 2 new replicas have been updated...
error: deployment "rag-app" exceeded its progress deadline
$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
rag-app-54d4bf4cdf-fhvfz   0/1     Running   0          61s
rag-app-7db4d9b489-dhdt5   1/1     Running   0          73s
rag-app-7db4d9b489-j8252   1/1     Running   0          73s
$ kubectl rollout undo deployment/rag-app
deployment.apps/rag-app rolled back
deployment "rag-app" successfully rolled out

Real output from the course's GitHub Actions run. The bad pod was running but never ready, so it got no traffic, and because maxUnavailable is 0 the two old pods were never removed. Users would have seen nothing; the rollout simply stalled until it hit its deadline and was undone.

A real example

Example

Serving a model on GPU nodes

A company running an open model with vLLM moved from a hand-managed GPU server to a managed Kubernetes cluster with a GPU node pool. The model server ran as a Deployment requesting one GPU per pod, with a readiness probe that only passed once weights were loaded (three minutes), a PodDisruptionBudget so maintenance never took both replicas down, and an autoscaler on queue length. A bad image update that failed to load weights stalled exactly like the run above: the old pods kept serving and the on-call engineer ran kubectl rollout undo from their phone.

The easy way

Start here

Use a managed Kubernetes service only when you need it. For learning, run kind or minikube locally and apply a Deployment and Service like the one above. For a single API, a managed container platform is usually simpler (serverless and managed hosting).

When you grow: the production setup

Platform teams use managed clusters with infrastructure as code (IaC), Helm or Kustomize for manifests, GitOps tools to sync from git, namespaces and RBAC per team, resource quotas, GPU node pools with taints, autoscaling for pods and nodes, network policies, secrets from a secrets manager, observability stacks, and progressive delivery controllers for canary releases (release strategies).

Trade-offs

What you gain

Self-healing, rolling updates and rollbacks built in.
Runs many services and GPU workloads on shared machines.
Portable across clouds; huge ecosystem.

What it costs you

Significant complexity and operational cost.
Many concepts and YAML to manage.
Misconfigured probes or resources cause subtle outages.

When not to use it

One or two small services that a managed container platform can run with far less effort.

Common mistakes

No readiness probe, so traffic hits pods before they can serve.
No resource requests, so pods get evicted or starve each other.
Liveness probes that call external APIs, causing restart storms.
Using :latest image tags, making rollbacks meaningless.
Editing live resources with kubectl edit instead of changing the YAML in git.

Interview questions

Try answering out loud first, then open the model answer.

Explain pods, Deployments and Services.

Model answer

A pod is the smallest deployable unit: one or more containers with shared networking and storage. A Deployment declares how many identical pods to run from a template and manages their lifecycle, including rolling updates and rollbacks. A Service provides a stable virtual address and DNS name that load-balances across the pods matching its selector that are ready.

How does Kubernetes protect users during a bad deployment?

Model answer

Readiness probes keep traffic away from new pods until they're ready; a rolling update strategy with maxUnavailable 0 keeps old pods serving until replacements are ready; progressDeadlineSeconds marks stalled rollouts as failed; and kubectl rollout undo (or automated rollback by a delivery controller) restores the previous revision.

What's special about running LLM inference on Kubernetes?

Model answer

GPU scheduling (resource requests for GPUs, node pools, taints and tolerations), long start-up times while weights load (startup and readiness probes, pre-cached weights), large memory needs, autoscaling on queue depth or KV-cache use rather than CPU, and careful rollouts because each replica is expensive.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Kubernetes: A system that runs containers across a cluster of machines, keeping the desired number of healthy copies running, updating them gradually and rolling back.
Pod: The smallest unit Kubernetes runs, made of one or more containers that share a network address and are scheduled together.
Deployment (Kubernetes): A Kubernetes object that keeps a set number of identical pods running from a template and handles rolling updates and rollbacks.
Service (Kubernetes): A stable name and address in Kubernetes that load-balances traffic across the ready pods matching a label.
Rolling update: Replacing old copies of an app with new ones a few at a time, keeping enough healthy copies serving throughout.

Before and after

Read first