Skip to content

All lessons

Area 07 // DeploymentLesson 120 / 14015 min read

Canary, blue-green and feature flags

Release strategies limit the damage of a bad release, a canary sends a small share of users to the new version first, blue-green keeps two full environments so you can switch traffic back instantly, and feature flags turn features on or off for chosen users without deploying.

Think of it like this

Analogy

A restaurant testing a new recipe first serves it to a few tables (canary). A theatre with two identical stages can move the show to the second stage and back in seconds if the lights fail (blue-green). And the chef can add a dish to the menu but keep it hidden until the kitchen is ready, offering it only to regulars at first, and pulling it instantly if something's wrong (feature flags).

The problem it solves

Even with tests and eval gates, some problems only show up with real users: a prompt change that annoys people, a model that's slower under real load, a retrieval tweak that fails on rare questions. If every release goes to 100% of users at once, every mistake hits everyone. You want to expose changes gradually, measure, and back out instantly.

How it works: canary releases

  1. Deploy the new version alongside the current one.
  2. Route a small share (1–5%) of users to it; bucket by a hash of the user ID so the same user stays on the same version (see the code).
  3. Compare metrics between canary and stable: errors, p95 latency and TTFT, cost per request, and AI quality signals: thumbs-down rate, online eval scores on sampled answers, guardrail blocks (alerts and dashboards).
  4. Ramp up in steps (5% → 25% → 50% → 100%) if metrics hold, or roll back automatically if they don't (rollbacks).
  5. Shadow mode (a variant): send a copy of real traffic to the new model or prompt without showing users its answers, and compare offline.

How it works: blue-green deployments

  1. Two identical environments: blue (live) and green (idle).
  2. Deploy the new version to green, run smoke tests and evals against it.
  3. Switch all traffic from blue to green at the load balancer or DNS.
  4. Keep blue running for a while: switching back is instant.
  5. Watch out: database and index changes must work with both versions (backward-compatible migrations), and you briefly pay for double capacity.

How it works: feature flags

  1. Wrap new behaviour in a flag checked at runtime: if is_on("new-reranker", user).
  2. Target: internal staff first, then a percentage, specific tenants, or regions.
  3. Change flags without deploying, from a flag service or config (open-source and hosted flag services exist, and a database table works to start).
  4. Kill switches for risky features (agent actions, a new model): turn off in seconds during an incident.
  5. Experiments: flags also power A/B tests comparing prompts or models on real outcomes.
  6. Clean up flags once fully rolled out; old flags become confusing technical debt.
A canary ramps up only while its metrics stay as good as the stable version's.
Loading the playground…

See it in code

Needs
Nothing: plain Python 3.

Stable user bucketing for a canary, plus feature flags with targeting and a kill switch.

canary_and_flags.py
import hashlib
 
def bucket(user_id: str, salt: str) -> float:
    """Stable 0-100 number per user: the same user always lands in the same bucket."""
    return int(hashlib.sha256(f"{salt}:{user_id}".encode()).hexdigest()[:8], 16) / 0xFFFFFFFF * 100
 
# CANARY: send a small, stable percentage of users to the new version.
def version_for(user_id: str, canary_percent: float) -> str:
    return "v2 (canary)" if bucket(user_id, "release-v2") < canary_percent else "v1 (stable)"
 
users = [f"user{i}" for i in range(1000)]
for pct in [5, 25, 100]:
    on_canary = sum(version_for(u, pct).startswith("v2") for u in users)
    print(f"canary {pct:3}%: {on_canary:4} of 1000 users on v2; user42 gets {version_for('user42', pct)}")
 
# FEATURE FLAGS: turn features on per user group without a deploy, and off instantly if something breaks.
FLAGS = {"new-reranker": {"enabled": True, "percent": 20, "allow": {"staff"}},
         "agent-mode":   {"enabled": False, "percent": 100, "allow": set()}}      # kill switch: enabled=False
def is_on(flag: str, user_id: str, groups: set) -> bool:
    f = FLAGS[flag]
    return f["enabled"] and (bool(groups & f["allow"]) or bucket(user_id, flag) < f["percent"])
print("new-reranker for staff user7:", is_on("new-reranker", "user7", {"staff"}))
print("agent-mode for anyone       :", is_on("agent-mode", "user7", {"staff"}))
Output
canary   5%:   57 of 1000 users on v2; user42 gets v1 (stable)
canary  25%:  254 of 1000 users on v2; user42 gets v1 (stable)
canary 100%: 1000 of 1000 users on v2; user42 gets v2 (canary)
new-reranker for staff user7: True
agent-mode for anyone       : False

Users who are on v2 at 5% stay on v2 at 25%, because their bucket number never changes. Using a different salt per flag keeps experiments independent.

A real example

Example

The prompt that only some users hated

A support assistant's new prompt passed every offline eval. It went out as a 5% canary. Errors, latency and cost were identical, but the thumbs-down rate on the canary was 2.3× the stable version's, concentrated in one language: the new prompt made answers in Spanish noticeably more formal. The automated canary analysis held the rollout; the team fixed the prompt, added Spanish examples to the golden set, and the second canary ramped to 100% in a day. Meanwhile a separate kill-switch flag let them disable the assistant's new "cancel subscription" action within seconds when a bug appeared, without touching the deployment.

The easy way

Start here

Use your platform's traffic splitting (many managed platforms can route a percentage to a new revision) for canaries, keep the previous revision ready to switch back, and add a simple flags table or config file with a kill switch for each risky AI feature.

When you grow: the production setup

Teams run progressive delivery with automated canary analysis (statistical comparison of error, latency, cost and quality metrics), blue-green for infrastructure-level changes, a flag service with targeting, audit history and experiments, shadow traffic for model swaps, and a policy that every risky feature ships behind a flag with an owner and an expiry date.

Trade-offs

What you gain

  • Bad releases affect few users and are reversed quickly.
  • Real-user quality signals before full rollout.
  • Flags decouple deploying from releasing.

What it costs you

  • Two versions running at once complicates data and debugging.
  • Blue-green doubles capacity during switches.
  • Too many flags create confusing code paths.

When not to use it

  • Tiny internal tools with a handful of users: a simple deploy with quick rollback is enough.

Common mistakes

  • Random per-request routing, so users flip between versions.
  • Judging canaries only on errors and latency, not quality and cost.
  • Canaries too small to show statistically meaningful differences.
  • Database changes that break the old version during blue-green.
  • Never removing old flags.

Interview questions

Try answering out loud first, then open the model answer.

How would you roll out a new prompt or model safely?

Model answer

Pass offline evals first; optionally run shadow traffic; release as a canary to a small, stable percentage of users bucketed by user ID; compare errors, latency, cost and quality signals (feedback, online evals, guardrail blocks) against stable; ramp gradually with automated analysis; keep a flag or routing switch for instant rollback; and record versions in traces.

Canary vs blue-green?

Model answer

A canary shifts a small, growing share of traffic to the new version and compares metrics, limiting blast radius. Blue-green runs two full environments and switches all traffic at once, giving instant rollback but no gradual exposure. Canaries suit catching subtle issues; blue-green suits fast, clean switchovers. They can be combined.

What are feature flags good for in AI products?

Model answer

Shipping code dark and enabling it for staff, tenants or percentages; A/B testing prompts and models; kill switches for risky capabilities (agent actions, new tools) during incidents; and regional or plan-based features, all without redeploying.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01What is a canary release?
  2. 02What's the main advantage of blue-green deployments?
  3. 03Why hash the user ID to choose canary users?
  4. 04What is a kill switch?
  5. 05Which metric is AI-specific when judging a canary?

0 of 5 answered

Key terms

Canary release
Sending a small share of users to a new version first and comparing its metrics with the stable version before rolling out to everyone.
Blue-green deployment
Running two identical environments and switching all traffic from the old one to the new one, so switching back is instant.
Feature flag
A setting checked at runtime that turns a feature on or off for chosen users without a new deploy.
Kill switch
A feature flag used to turn a risky feature off instantly during an incident.
Rollback
Returning to the previous known-good version of code, config, prompt or model after a bad release.

Before and after

Progress is saved in this browser only.