Area 07 // DeploymentLesson 111 / 14014 min read

CI/CD pipelines

A CI/CD pipeline is an automated sequence of steps that runs on every change, testing it (continuous integration) and then building and releasing it (continuous delivery or deployment), so that only changes which pass every check ever reach users.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A car factory's assembly line has inspection stations: check the parts, then the engine, then the brakes, then a test drive. A car that fails any station is pulled off the line; it never reaches the showroom. Cheap checks come first so you don't install seats in a car with a faulty engine. A CI/CD pipeline is that line for software.

The problem it solves

Manual releases are slow and error-prone: someone forgets to run the tests, builds from the wrong branch, or skips the eval because it's late. AI apps add new ways to break things silently: a prompt tweak that lowers answer quality, a retrieval change that hurts recall, a model switch that doubles cost. You want every change checked the same way, automatically, and released only if it passes.

How it works, step by step

Trigger: a push or pull request starts the pipeline (Git and GitHub).
Fast checks first: formatting and linting, type checks, secret scanning (seconds).
Unit and integration tests with mock models, so they're fast, free and deterministic.
Eval gate: run the golden set; block if quality metrics fall below thresholds or regress versus main (eval gates in CI).
Build the artifact once: a container image tagged with the commit SHA (Docker), scanned for vulnerabilities, pushed to a registry.
Deploy to staging automatically; run smoke tests against it (health, a few real requests).
Promote to production: the same image, automatically (continuous deployment) or after an approval (continuous delivery), often gradually (release strategies).
Verify and roll back: watch health, errors, latency and cost after the release; roll back automatically if they break (health checks and rollbacks).
Report: status checks on the PR, notifications, and a record of what was deployed where.

A typical pipeline for an AI app. Any failure stops everything after it.

See it in code

Needs: Nothing: plain Python 3. Each stage is a stand-in; the real pipeline is the workflow file in the GitHub Actions lesson.

A pipeline as code: stages in order, stop at the first failure. The second commit breaks retrieval quality and never gets near production.

pipeline.py

# A CI/CD pipeline is a list of stages; each must pass before the next runs. Fail fast, cheapest first.
def lint():        return True, "ruff: 0 problems"
def unit_tests():  return True, "7 passed in 1.9s"
def eval_gate():   return (RECALL >= 0.90), f"recall@3 = {RECALL:.2f} (threshold 0.90)"
def build():       return True, "image rag-app:3f9c2ab built (212 MB)"
def deploy_stg():  return True, "staging ready after 4s"
def deploy_prod(): return True, "canary 5% -> 100%"
 
PIPELINE = [("lint", lint), ("unit tests", unit_tests), ("eval gate", eval_gate),
            ("build image", build), ("deploy staging", deploy_stg), ("deploy prod", deploy_prod)]
 
def run(commit: str):
    print(f"pipeline for commit {commit}")
    for name, stage in PIPELINE:
        ok, detail = stage()
        print(f"  {'PASS' if ok else 'FAIL'}  {name:15} {detail}")
        if not ok:
            print("  stopped: nothing after this stage runs, production is untouched")
            return
 
RECALL = 0.96
run("3f9c2ab")
RECALL = 0.83                      # someone changed the chunking and retrieval got worse
run("8d1e004")

Output

pipeline for commit 3f9c2ab
  PASS  lint            ruff: 0 problems
  PASS  unit tests      7 passed in 1.9s
  PASS  eval gate       recall@3 = 0.96 (threshold 0.90)
  PASS  build image     image rag-app:3f9c2ab built (212 MB)
  PASS  deploy staging  staging ready after 4s
  PASS  deploy prod     canary 5% -> 100%
pipeline for commit 8d1e004
  PASS  lint            ruff: 0 problems
  PASS  unit tests      7 passed in 1.9s
  FAIL  eval gate       recall@3 = 0.83 (threshold 0.90)
  stopped: nothing after this stage runs, production is untouched

Note the second commit passed every unit test: only the eval gate noticed that answers would get worse. That's why AI pipelines need it.

A real example

Example

From Friday-evening releases to twenty a day

A small team released their AI writing tool once a week, by hand, on Friday evenings; about one release in four caused an incident. They built a pipeline: lint and tests in under two minutes, an eval gate on 150 golden examples using a cheap judge model, an image build, automatic staging deploys with smoke tests, and a one-click production promotion with a 10% canary. Releases became small and boring: about twenty a week, with incidents rare and rolled back in minutes.

The easy way

Start here

Start with one workflow that runs your tests and eval script on every pull request and blocks merging if they fail. Add the image build and an automatic staging deploy next; production can stay a manual button for a while.

When you grow: the production setup

Mature pipelines include caching and parallel jobs for speed, required status checks, image signing and vulnerability scanning, environment protection rules and approvals, progressive delivery with automated canary analysis, deployment records linked to commits and traces, cost and latency budgets as gates, and pipeline definitions reviewed like code.

Trade-offs

What you gain

Every change gets the same checks.
Small, frequent, low-risk releases.
Quality regressions are caught before users see them.

What it costs you

Setup and maintenance effort.
Slow or flaky pipelines frustrate developers.
Eval stages cost money (model calls) on every run.

When not to use it

Even a solo project benefits from CI on pull requests; skip automated production deploys only if releases are rare and risky.

Common mistakes

Rebuilding the image for production instead of promoting the tested one.
Calling real models in unit tests (slow, flaky, costly).
No eval gate, so quality regressions ship.
Allowing merges when checks fail ("just this once").
Flaky tests everyone learns to ignore.

Interview questions

Try answering out loud first, then open the model answer.

Design a CI/CD pipeline for an LLM application.

Model answer

On every PR: lint, type checks and secret scanning; unit tests with mocked models; an eval gate on a golden set comparing against thresholds and main; build and scan an image tagged with the SHA. On merge: push the image, deploy to staging, run smoke tests and evals against staging, then promote the same image to production via canary with automated monitoring and rollback. Track prompt and model versions as part of the release.

How do you keep evals in CI affordable and stable?

Model answer

Use a focused golden set for PRs and a larger nightly suite; cache model responses where inputs haven't changed; use cheaper judge models where they agree with humans; fix sampling settings where supported; run multiple samples for noisy metrics; and compare against the main branch with a tolerance rather than exact equality.

Continuous delivery vs continuous deployment?

Model answer

Continuous delivery means every change that passes the pipeline is releasable, but a person decides when to deploy to production. Continuous deployment releases every passing change to production automatically. Many teams auto-deploy to staging and use approval or automated canary analysis for production.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

CI/CD: Continuous integration and continuous delivery. Automatically building, testing and deploying every change through a pipeline.
Continuous integration (CI): Automatically testing and checking every change when it is pushed or proposed, so problems are found within minutes.
Continuous delivery (CD): Keeping every change that passes the pipeline ready to release. Continuous deployment goes further and releases it automatically.
Pipeline (CI/CD): An automated sequence of steps (checks, tests, evals, build, deploy) that runs on every change; a failure stops the steps after it.
Artifact (build): The packaged output of a build, such as a container image, that is tested once and then deployed unchanged to each environment.
Eval gate: A pipeline step that runs evals on a change and blocks the merge or release if quality falls below thresholds or regresses versus main.

Before and after

Read first