Skip to content

All lessons

Area 07 // DeploymentLesson 123 / 14014 min read

Alerts and dashboards

Dashboards show how your AI app is doing at a glance (traffic, errors, latency, cost and quality), and alerts notify a person only when something needs action, ideally based on service-level objectives and how fast the error budget is being used up.

Think of it like this

Analogy

A car dashboard shows speed, fuel and temperature at a glance. Warning lights stay off unless something needs attention, and a flashing red oil light means "stop soon", while a service reminder means "book it this month". A dashboard full of flashing lights for normal things would be worse than useless: you'd stop looking.

The problem it solves

Once your app is live, things go wrong at 3 a.m.: a provider slows down, a deploy breaks retrieval, a bot runs up your bill, quality slips after a model update. If you only find out from angry users, you're too late. But alerting on every small blip burns out whoever's on call until they ignore alerts entirely. You need clear dashboards and a few alerts that fire only when action is needed.

How it works, step by step

  1. Collect metrics and traces: requests, errors, latency, tokens, cost, model and prompt versions (tracing, latency and cost metrics, logging).
  2. Build dashboards around questions people ask:
    • Golden signals: traffic, error rate, latency (p50, p95, time to first token), saturation (queue depth, GPU memory).
    • AI signals: tokens and cost per hour, per feature and per tenant; cache hit rate; provider errors and 429s; guardrail block rate; feedback (thumbs-down rate); online eval scores.
    • Release markers so you can see if a change lines up with a deploy.
  3. Define SLOs for what users care about: e.g. 99.5% of requests succeed, 95% of first tokens arrive within 2 s. The error budget is what's left (0.5%).
  4. Alert on burn rate: how fast you're spending the budget. Page when both a long window (1 h) and a short window (5 min) burn fast, so the problem is both significant and still happening; open a ticket for slower burns (see the code).
  5. Add a few direct alerts for things SLOs miss: cost per hour above twice normal (denial of wallet), provider rate-limit storms, quality score drops, failed batch jobs.
  6. Route by urgency: pages for urgent and user-facing; tickets or chat messages for the rest; dashboards for everything else.
  7. Every alert gets a runbook: what it means, the dashboard link, first steps (check recent deploys, roll back: rollbacks).
  8. Review alerts regularly: delete noisy ones, add missing ones after incidents.

See it in code

Needs
Nothing: plain Python 3. Monitoring tools (Prometheus and Grafana, cloud monitoring, Datadog and others) evaluate rules like these continuously.

Multi-window burn-rate alerting for a 99.5% SLO: a slow leak becomes a ticket, a bad deploy pages someone, a recovered incident doesn't.

burn_rate_alerts.py
# Alerting on an SLO with burn rates: page a human only when the error budget is being used up fast.
SLO = 0.995                         # 99.5% of requests succeed (errors = 5xx, timeouts, guardrail failures)
BUDGET = 1 - SLO                    # 0.5% of requests may fail over 30 days
 
def burn_rate(errors: int, total: int) -> float:
    return (errors / total) / BUDGET    # 1.0 = using the budget exactly as fast as allowed
 
def decide(windows: dict) -> str:
    rates = {w: burn_rate(e, t) for w, (e, t) in windows.items()}
    summary = ", ".join(f"{w} burn {r:.1f}x" for w, r in rates.items())
    # Multi-window rule: page if BOTH the 1h and 5m windows burn faster than 14.4x
    # (that pace spends 2% of the 30-day budget in one hour); open a ticket if slower but sustained.
    if rates["1h"] > 14.4 and rates["5m"] > 14.4:
        return f"{summary} -> PAGE on-call now"
    if rates["1h"] > 3:
        return f"{summary} -> TICKET for working hours"
    return f"{summary} -> OK"
 
# (errors, total requests) per window, from the metrics system
print("slow leak :", decide({"1h": (720, 36_000), "5m": (50, 3_000)}))
print("bad deploy:", decide({"1h": (2_900, 36_000), "5m": (310, 3_000)}))
print("recovered :", decide({"1h": (2_900, 36_000), "5m": (4, 3_000)}))
 
# LLM-specific signals deserve their own alerts: cost per hour, p95 time to first token, eval score drift.
cost_per_hour, normal = 182.0, 60.0
print("cost alert:", cost_per_hour > 2 * normal, f"(${cost_per_hour:.0f}/h vs normal ${normal:.0f}/h)")
Output
slow leak : 1h burn 4.0x, 5m burn 3.3x -> TICKET for working hours
bad deploy: 1h burn 16.1x, 5m burn 20.7x -> PAGE on-call now
recovered : 1h burn 16.1x, 5m burn 0.3x -> TICKET for working hours
cost alert: True ($182/h vs normal $60/h)

After the bad deploy was rolled back, the 5-minute window recovered, so nobody gets woken up for a problem that's already over; the 1-hour window still files a ticket so it gets reviewed.

A real example

Example

From 40 alerts a week to 3

An AI team's on-call engineer got about 40 alerts a week (CPU above 70%, single 500 errors, every provider timeout) and had started muting them. During one muted night, a broken deploy failed a third of requests for two hours.

They rebuilt alerting around two SLOs (success rate and time to first token) with burn-rate pages, plus three direct alerts: cost per hour above twice normal, provider 429 rate above 5%, and the hourly online eval score dropping below 0.8. Each alert linked to a dashboard row and a runbook starting with "check the latest deploy; roll back if it's recent". Alerts fell to about 3 a week, all actionable, and the next bad deploy paged within 6 minutes.

The easy way

Start here

Make one dashboard with requests, error rate, p95 latency, time to first token and cost per hour, with deploy markers. Add three alerts: error rate high for 10 minutes, cost per hour above twice normal, and the app's /ready failing. Link each alert to a short runbook.

When you grow: the production setup

Teams define SLOs per user journey, alert on multi-window burn rates, add AI-specific alerts (cost, quality drift, guardrail and refusal spikes, provider health), route by severity with on-call rotations, keep runbooks next to alerts, annotate dashboards with releases and flag changes, and review alert quality after incidents.

Trade-offs

What you gain

  • Problems are found in minutes, not from user complaints.
  • SLO-based alerts focus on user impact, cutting noise.
  • Dashboards make releases and costs visible to everyone.

What it costs you

  • Takes effort to choose good SLOs and thresholds.
  • Monitoring tools and metric storage cost money.
  • Quality metrics are harder to measure than errors.

When not to use it

  • Don't page anyone for a hobby project; dashboards and email alerts are enough.

Common mistakes

  • Alerting on causes (CPU, memory) instead of symptoms users feel.
  • Paging for things nobody can act on.
  • No cost alerts, so spend spikes surface on the invoice.
  • Alerts without runbooks or dashboard links.
  • Dashboards without deploy markers, hiding the most common cause of incidents.

Interview questions

Try answering out loud first, then open the model answer.

What would you monitor for an LLM application in production?

Model answer

Golden signals (traffic, error rate, latency including TTFT and time per token, saturation such as queue depth or GPU memory) plus AI signals: tokens and cost by feature and tenant, cache hit rates, provider errors and rate limits, guardrail blocks and refusals, user feedback, online eval scores and drift, all broken down by model and prompt version with release markers.

Explain SLOs, error budgets and burn-rate alerts.

Model answer

An SLO is a target for a user-facing metric (e.g. 99.5% success over 30 days). The error budget is the allowed failure (0.5%). Burn rate is how fast you're consuming it relative to plan; a burn rate of 14.4 sustained for an hour uses 2% of a 30-day budget. Alerting when both a long and a short window exceed a high burn rate pages only for significant, ongoing problems; slower burns open tickets.

How do you avoid alert fatigue?

Model answer

Alert on user-facing symptoms via SLOs rather than every cause; page only for urgent, actionable issues; use multi-window conditions; route lower-severity issues to tickets; attach runbooks; and regularly delete or tune alerts that fire without action.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01What is an error budget?
  2. 02What does a burn rate of 10x mean?
  3. 03Why use both a long and a short window for alerts?
  4. 04Which is an AI-specific metric worth a dashboard panel?
  5. 05When should an alert page someone at night?

0 of 5 answered

Key terms

SLO (service level objective)
A target for a user-facing metric over a period, such as 99.5% of requests succeeding over 30 days.
Error budget
The amount of failure an SLO allows, such as 0.5% of requests for a 99.5% target.
Burn rate
How fast the error budget is being used compared with the planned pace; 1x is exactly on budget, 10x would use a month's budget in three days.
Alert
An automatic notification when a monitored condition needs attention, either paging someone urgently or creating a ticket.
Dashboard
A page of charts showing a system's key metrics, such as traffic, errors, latency, cost and quality, at a glance.
On-call
A rota of engineers who respond to urgent alerts outside normal hours.

Before and after

Read first

Read next

Progress is saved in this browser only.