Skip to content
← All cheat sheets

Cheat sheet · Area 07

Deployment

Git and GitHub for deployment

Git records every change to your code as a commit, branches let you work on changes safely, and GitHub hosts the repository so teammates can review pull requests and automation can test and deploy each change, which makes git history the backbone of every deployment.

  • Commit = a saved snapshot with a message. Branch = a separate line of work. Merge = bring it back.
  • Flow: branch → commits → push → pull request → review + CI → merge to main → deploy.
  • Protect main: required reviews and passing checks; never push straight to it.
  • Tags (v1.2.0) mark releases; deploys reference a commit SHA or tag so you know exactly what's running.
  • Never commit secrets, .env files, large model weights or datasets (use .gitignore, Git LFS or storage).

What deploying means, and environment configs

Deploying means taking a specific version of your app from your laptop to servers where real users can reach it, usually through separate environments (development, staging and production) that run the same code with different configuration supplied from outside the code.

  • Deploy = build a version → put it on servers → start it with the right config → send traffic → watch it.
  • dev (your laptop) → staging (production-like, for testing) → prod (real users).
  • Same code and image everywhere; only CONFIG differs (env vars, secrets, URLs, model names, limits).
  • Config comes from the environment, validated at start-up; fail fast if something's missing.
  • Never test on prod data you don't need; never use prod keys in dev.

Docker

Docker packages your app together with its exact Python version, libraries and files into an image, a portable bundle that runs the same way as a container on your laptop, in CI and on any server, which removes "works on my machine" problems from deployment.

  • Dockerfile = recipe. Image = the built, read-only package. Container = a running copy of an image.
  • Order layers from least to most often changed: base image → dependencies → code, so rebuilds reuse the cache.
  • Use slim base images, a non-root user, a HEALTHCHECK, and .dockerignore; never bake secrets into images.
  • Tag images with the git SHA; push to a registry (GHCR, Docker Hub, ECR...); deploy that exact tag.
  • docker build -t app:v1 . · docker run -p 8000:8000 app:v1 · docker ps · docker logs

Docker Compose

Docker Compose describes all the containers an app needs (the API, a database, a cache, a local model server) in one compose.yaml file, so a single command starts the whole stack with the right ports, settings, volumes and health checks, on a laptop, in CI or on a small server.

  • compose.yaml lists services (containers), their image or build, ports, environment, volumes and dependencies.
  • docker compose up -d --wait starts everything and waits until health checks pass; docker compose down stops it.
  • Services reach each other by name on a private network (e.g. http://ollama:11434).
  • Volumes keep data (databases, downloaded models) when containers are recreated.
  • Profiles turn optional services on: docker compose --profile local-llm up.

CI/CD pipelines

A CI/CD pipeline is an automated sequence of steps that runs on every change, testing it (continuous integration) and then building and releasing it (continuous delivery or deployment), so that only changes which pass every check ever reach users.

  • CI: on every push/PR, automatically lint, test, evaluate and build. CD: automatically release what passed.
  • Order stages cheapest and fastest first; stop at the first failure.
  • Build once, deploy the same artifact (image) to staging then production.
  • AI pipelines add: eval gates, prompt/model version checks, cost checks, guardrail regression tests.
  • Continuous delivery = ready to deploy at the push of a button; continuous deployment = deploys automatically.

GitHub Actions

GitHub Actions runs automated workflows (tests, eval gates, image builds, deployments) on GitHub's machines whenever something happens in your repository, such as a push or pull request, using YAML files in .github/workflows that describe jobs made of steps.

  • A workflow = a YAML file in .github/workflows/ with triggers (on), jobs and steps.
  • Jobs run on fresh runner machines (ubuntu-latest); steps are shell commands (run) or reusable actions (uses).
  • needs: chains jobs into a pipeline; a failed job skips everything that depends on it.
  • Secrets live in repo/environment settings and arrive as ${{ secrets.NAME }}; GITHUB_TOKEN is built in.
  • Make the checks required in branch protection so red runs block merges.

Eval gates in CI

An eval gate is a pipeline step that runs your evaluation set against the new version and blocks the merge or release if quality metrics fall below fixed thresholds or drop compared with the current main branch, so prompt, model and retrieval changes can't silently make answers worse.

  • Gate = evals + rules + exit code. Non-zero exit fails the job and blocks the merge.
  • Two rules: absolute floors (recall@3 ≥ 0.90) AND no regression vs main (drop ≤ 0.02).
  • Trigger on changes to prompts, models, retrieval, chunking, tools, guardrails, not just code.
  • Keep PR gates fast and cheap; run the big suite nightly.
  • Publish the scores in the PR so reviewers see the trade-off (quality, cost, latency).

Infrastructure as code

Infrastructure as code means describing servers, databases, networks, buckets and permissions in version-controlled files that a tool such as OpenTofu or Terraform turns into real cloud resources, so environments are reproducible, reviewed like code and changed by plan and apply instead of by clicking in a console.

  • Describe the desired infrastructure in files; the tool works out what to create, change or delete.
  • init (download providers) → plan (preview changes) → apply (make them) → state remembers what exists.
  • Review plans in pull requests; apply from CI, not laptops.
  • Keep state remote, locked and private (it can contain secrets).
  • Tools: OpenTofu/Terraform (HCL), Pulumi (real languages), AWS CDK, cloud-specific templates.

Serverless functions and the easy hosting route

Serverless functions run your code only when a request arrives and bill per use, while managed platforms (container hosts, Hugging Face Spaces and similar) run your app from a repo or image with little setup, which makes them the fastest way to ship most AI apps that call model APIs.

  • Serverless function: a handler called per request; scales to zero; pay per invocation and duration.
  • Watch: cold starts, time limits, no local state, streaming support, concurrency limits.
  • Managed container platforms run your Docker image with autoscaling (e.g. Cloud Run, Fly.io, Render, Railway).
  • Hugging Face Spaces: free demo hosting for Gradio/Streamlit/Docker apps, with paid GPU upgrades.
  • Great when you call model APIs; self-hosted models usually need GPU hosts instead.

Ollama for local models

Ollama downloads open models and runs them on your own laptop or server with one command, exposing a simple local HTTP API, which is ideal for offline development, privacy-sensitive experiments and cheap testing, as long as you evaluate whether a small local model is actually good enough for the task.

  • ollama pull <model> downloads it; ollama run <model> chats; the API is on http://localhost:11434.
  • Popular small models (as of Nov 2026): llama3.2, qwen3, gemma3; embeddings: nomic-embed-text.
  • Models come quantized (often 4-bit), so a 0.6B model is ~0.5 GB and an 8B one ~5 GB.
  • CPU works for small models; a GPU (or Apple silicon) makes bigger ones usable.
  • Small local models fail more often: run your evals before relying on one.

GPU instances

A GPU instance is a rented cloud machine with one or more GPUs, used to run or fine-tune open models yourself; choosing one means checking that the model fits in GPU memory, estimating throughput and cost per token, and comparing that with simply calling a model API.

  • Step 1, memory: params × bytes per param × ~1.2 + KV cache must fit in VRAM.
  • FP16 = 2 bytes/param, INT8 = 1, INT4 = 0.5. An 8B model: ~19 GB at FP16, ~5 GB at INT4 (+ KV cache).
  • Step 2, throughput: tokens/s with batching (vLLM) → cost per 1M tokens = $/hour ÷ tokens per hour × 1M.
  • Self-hosting only wins if the GPU stays busy; idle GPUs are expensive.
  • Options: on-demand, reserved, spot (cheap, can be taken away), serverless GPU platforms.

Kubernetes basics

Kubernetes runs containers across a cluster of machines and keeps them in the state you describe, the right number of healthy copies, reachable through a stable address, updated gradually and rolled back when a new version can't get ready, which makes it a common platform for serving AI apps and models at scale.

  • Pod = one or more containers scheduled together. Deployment = keeps N identical pods running and updates them.
  • Service = a stable name and address that load-balances across the ready pods.
  • Probes: liveness restarts stuck containers; readiness decides who gets traffic.
  • Rolling update with maxUnavailable 0: new pods must be ready before old ones go; kubectl rollout undo rolls back.
  • kubectl apply -f · get pods · rollout status · logs · describe · scale

Autoscaling and load balancing

Autoscaling adds or removes copies of your service as demand changes, and a load balancer spreads incoming requests across those copies, so an AI app stays fast at peak without paying for idle capacity, as long as you scale on the right signal and balance by real load.

  • Horizontal scaling = more copies (replicas); vertical = bigger machines.
  • Kubernetes HPA: desired = ceil(current × metric / target), within min and max.
  • Scale on the right signal: CPU for API servers; queue depth, concurrent requests or KV-cache use for LLM servers.
  • Load balancing: round robin is fine for equal requests; least-outstanding-requests suits uneven LLM calls.
  • Health checks keep traffic away from unready replicas; scale-down needs a cool-down to avoid flapping.

Canary, blue-green and feature flags

Release strategies limit the damage of a bad release, a canary sends a small share of users to the new version first, blue-green keeps two full environments so you can switch traffic back instantly, and feature flags turn features on or off for chosen users without deploying.

  • Canary: 1–5% of traffic to the new version, compare metrics, then ramp to 100% or roll back.
  • Blue-green: two identical environments; switch traffic all at once; switching back is instant.
  • Feature flags: ship code dark, enable per user/group/percentage, kill switch to turn off instantly.
  • For AI: compare quality signals too (feedback, eval scores on live samples, guardrail blocks, cost per request).
  • Bucket users stably (hash of user ID) so each person gets a consistent experience.

Prompt versioning, model versioning and registries

Prompts, model choices and fine-tuned weights change an AI app's behaviour as much as code does, so each needs an immutable version, a record of how it was evaluated, and a registry with labels like "production" that you can move forward to release or back to roll back.

  • Version = immutable snapshot (template + model + parameters), identified by a number or content hash.
  • Labels/aliases (production, staging) are movable pointers; rollback = move the label back.
  • Pin exact model IDs/snapshots; aliases like 'latest' can change under you.
  • Log the prompt and model version on every trace so you can explain any answer.
  • Registries: prompt management in tools like Langfuse; model registries like MLflow or Hugging Face Hub.

Health checks and rollbacks

Health checks are small endpoints that tell the platform whether an instance is alive (liveness) and ready for traffic (readiness), and rollbacks return production to the last good version quickly when a release fails those checks or its metrics get worse.

  • Liveness (/health): is the process alive? Fail → restart it. Keep it cheap; no external calls.
  • Readiness (/ready): can it serve now (index loaded, config and secrets present)? Fail → no traffic.
  • Deploys wait for readiness before sending traffic; a release that never gets ready is rolled back.
  • Rollback = redeploy the previous image/label. Practise it; keep migrations backward compatible.
  • Automate: roll back when error rate, latency or quality metrics breach thresholds after a release.

Alerts and dashboards

Dashboards show how your AI app is doing at a glance (traffic, errors, latency, cost and quality), and alerts notify a person only when something needs action, ideally based on service-level objectives and how fast the error budget is being used up.

  • Golden signals: traffic, errors, latency (p50/p95, TTFT), saturation. Add for AI: cost, tokens, quality, guardrail blocks.
  • SLO: target like '99.5% of requests succeed'; error budget = the 0.5% you may fail.
  • Alert on burn rate (how fast you spend the budget), using a long and a short window.
  • Page only for urgent, user-facing, actionable problems; everything else is a ticket or a dashboard.
  • Every alert links to a dashboard and a runbook.

Capstone: ship the RAG app

The deployment capstone takes the course's RAG app from a folder of Python to a shipped service, a Docker image, a GitHub Actions pipeline with tests and an eval gate, a staging deploy that waits for a health check, and an automatic rollback that is tested on every run, all with real runs from the course repository.

  • Pipeline: test → eval gate → build image + Compose smoke test → push (tag = git SHA) → deploy staging.
  • /health = alive (restart if not); /ready = can serve (index loaded, provider configured).
  • Deploy = start the new version next to the old, wait for /ready, then switch; not ready in 15 s → remove it, keep the old.
  • Rollback drill on every run: deploy a release that can't get ready and assert it's rejected.
  • Files: Dockerfile, compose.yaml, deploy/deploy.sh, k8s/, .github/workflows/rag-app.yml.