Health checks and rollbacks

How it works: health checks

1. Liveness (/health): returns 200 if the process can respond. Cheap, no external dependencies. If it fails repeatedly, the platform restarts the container.
2. Readiness (/ready): returns 200 only when the instance can serve real requests: index or model loaded, required config and secrets present, essential dependencies reachable. Otherwise 503, and the load balancer sends it no traffic (load balancing).
3. Startup checks: slow starters (model servers loading weights) get a startup probe or longer initial delay so they aren't killed while loading.
4. Wire them up: Docker HEALTHCHECK, Kubernetes livenessProbe and readinessProbe, managed platforms' health check path (Docker, Kubernetes).
5. Report the version in the response, so you can see what's running.
6. Deep checks (does a real question get a sensible answer?) belong in smoke tests and synthetic monitoring, not in liveness.

How it works: rollbacks

1. Keep previous versions deployable: immutable images tagged by commit, previous prompt and model labels (versioning).
2. Gate deploys on readiness: start the new version, wait for /ready, then shift traffic; if it never becomes ready, remove it and keep the old one (the capstone's deploy.sh does exactly this).
3. Watch after release: error rate, latency, cost, and quality signals compared with before (alerts).
4. Roll back automatically when thresholds are breached (see the code), or with one command (kubectl rollout undo, redeploying the previous tag, moving a label back).
5. Make data changes reversible: backward-compatible migrations (add columns first, remove later), versioned indexes kept until the new one is proven.
6. Practise: rollback drills in staging (the capstone pipeline runs one on every deploy).
7. Afterwards: a short incident review and a new test so it can't recur.

import os
from fastapi import FastAPI, Response
from fastapi.testclient import TestClient
 
INDEX = ["chunk-1", "chunk-2"]                      # pretend the vector index loaded at start-up
app = FastAPI()
 
@app.get("/health")                                 # LIVENESS: is the process alive? (restart if not)
def health():
    return {"status": "ok"}
 
@app.get("/ready")                                  # READINESS: can it serve real traffic right now?
def ready(response: Response):
    checks = {"index_loaded": len(INDEX) > 0,
              "api_key_present": bool(os.environ.get("ANTHROPIC_API_KEY"))}
    if not all(checks.values()):
        response.status_code = 503                  # load balancer stops sending traffic here
    return checks
 
client = TestClient(app)
print("health:", client.get("/health").status_code, client.get("/health").json())
r = client.get("/ready")
print("ready :", r.status_code, r.json())           # a bad deploy: secret not configured
os.environ["ANTHROPIC_API_KEY"] = "placeholder"
r = client.get("/ready")
print("ready :", r.status_code, r.json())
 
# ROLLBACK decision after a deploy: compare the new version's error rate with the old one's.
def should_roll_back(new_errors, new_total, old_rate, tolerance=0.01):
    return new_errors / new_total > old_rate + tolerance
print("roll back?", should_roll_back(new_errors=42, new_total=1_000, old_rate=0.005))

health: 200 {'status': 'ok'}
ready : 503 {'index_loaded': True, 'api_key_present': False}
ready : 200 {'index_loaded': True, 'api_key_present': True}
roll back? True