Area 05 // AI System DesignLesson 90 / 14018 min read

Case study: a chatbot for 1 million users

Designing a customer chatbot for a million daily users means estimating traffic and tokens, choosing models and a gateway, streaming answers, storing conversations, grounding answers with retrieval, and adding caching, guardrails, rate limits and monitoring, then working out what it will cost.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Planning a restaurant that will serve a million meals a day. Before picking ovens you count: how many guests, how many at the busiest hour, how long each meal takes. Then you design the kitchen line (stations, a manager routing orders, a quick counter for simple dishes), the pantry (stored recipes and customer notes), the hygiene checks and the budget. The numbers decide the design.

The problem it solves

"Design a chatbot for a million users" is a classic AI system design interview and a real project. The model call is the easy part. The hard parts are estimating load and cost, keeping answers fast and grounded, remembering conversations, staying up when a provider fails, keeping users safe, and not spending a million dollars a month.

How it works, step by step

Requirements: a customer-support chatbot on web and mobile; answers grounded in help articles; can look up the user's orders; streaming; available 99.9%; p95 time to first token under 1.5 s; supports several languages.
Estimate the load (see the code): messages per day, average and peak QPS, tokens per message, peak tokens per second. Say your assumptions out loud.
Architecture:
- Client sends messages and receives a stream over SSE.
- API layer: auth, per-user rate limits, request validation.
- Chat service (stateless, autoscaled): loads the conversation, assembles context (token budgets), retrieves help articles (RAG), and offers order-lookup tools scoped to the user (tool calling).
- Gateway: routing (small model for simple questions), fallbacks across providers, semantic cache for general questions, prompt caching for the stable prefix.
- Guardrails on input and output (guardrails architecture).
- Storage: conversations and messages in Postgres (databases), session state and rate limits in Redis, a vector index for articles.
- Async: summarising long conversations, analytics, and labelling for evals run in background jobs.
Reliability: multiple providers or regions, timeouts and retries, graceful degradation ("I'm having trouble, here's the help centre link").
Observability: traces per message (tracing), TTFT and cost dashboards, online evals on samples, user feedback buttons.
Cost: estimate per model choice (see the code), then apply the cost levers.

Architecture

A chatbot for 1M daily users: stateless services, a gateway in front of the models, data stores behind.

See it in code

Needs: Nothing: plain Python 3. Example prices as of November 2026 (mid-tier $2/$10, small $0.10/$0.50 per 1M tokens).

Back-of-the-envelope numbers for traffic, throughput and cost.

chatbot_estimate.py

# Back-of-the-envelope for a chatbot with 1M daily active users (state your assumptions out loud).
DAU = 1_000_000
msgs_per_user = 6                 # messages per user per day
peak_factor = 3                   # peak hour vs average hour
in_tokens, out_tokens = 3_000, 300   # per message (system + history + retrieved docs; reply)
 
msgs_per_day = DAU * msgs_per_user
avg_qps = msgs_per_day / 86_400
peak_qps = avg_qps * peak_factor
print(f"messages/day: {msgs_per_day:,}  avg QPS: {avg_qps:.0f}  peak QPS: {peak_qps:.0f}")
 
peak_out_tps = peak_qps * out_tokens                     # output tokens per second at peak
print(f"peak output tokens/s: {peak_out_tps:,.0f}")
 
for name, p_in, p_out, cached_share in [("mid-tier API", 2.00, 10.00, 0.6), ("small API", 0.10, 0.50, 0.6)]:
    cost_in = in_tokens * ((1 - cached_share) * p_in + cached_share * p_in * 0.1) / 1e6
    cost = (cost_in + out_tokens * p_out / 1e6) * msgs_per_day * 30
    print(f"{name:12}: ~${cost:,.0f}/month (60% of input from prompt cache)")

Output

messages/day: 6,000,000  avg QPS: 69  peak QPS: 208
peak output tokens/s: 62,500
mid-tier API: ~$1,036,800/month (60% of input from prompt cache)
small API   : ~$51,840/month (60% of input from prompt cache)

The 20× gap is the design decision: route most traffic to the small model, keep the mid-tier for hard questions, and add a semantic cache for common ones. 62,500 output tokens per second at peak also tells you to check the provider's rate limits (or GPU count if self-hosting) early.

A real example

Example

How the design plays out

A retailer launching this design sent about 75% of messages to a small model (order status, returns, opening hours), 25% to a mid-tier model (complaints, multi-step problems), and served 18% of general questions from a semantic cache. Monthly model spend came in around $230,000, well below the all-mid-tier estimate.

During a provider outage in month two, the gateway failed over to a second provider within seconds; users saw slightly different wording but no errors. Online evals on 1% of conversations caught a drop in grounding after a help-centre migration broke some article links, fixed within a day.

The easy way

Start here

For a first version: one model through a gateway with a fallback, streaming over SSE, conversations in Postgres, help articles in pgvector, rate limits and an output guard, and a cost dashboard. Measure real traffic before adding routing and caches.

When you grow: the production setup

At scale, teams add difficulty-based routing, semantic and prompt caching, multi-region deployment, per-tenant quotas if it's a platform, async summarisation of long conversations, online evals and feedback loops, A/B tests of prompts and models, canary releases (release strategies), and capacity agreements with providers.

Trade-offs

What you gain

Stateless services scale horizontally behind a load balancer.
A gateway isolates the app from provider outages and price changes.
Routing and caching keep the bill proportional to difficulty.

What it costs you

Many components to run and monitor.
Routing and caching need evals to stay safe.
Provider rate limits can cap growth without capacity planning.

When not to use it

An internal bot for 200 people doesn't need this: one service, one model, one database.

Common mistakes

Jumping to architecture without estimating traffic and cost.
Keeping conversation state in server memory, which breaks with several servers.
One model for everything, with no fallback.
No per-user rate limits, so one script can drain the budget.
Ignoring TTFT and measuring only total latency.

Interview questions

Try answering out loud first, then open the model answer.

Design a chatbot for 1 million daily users.

Model answer

Clarify requirements (purpose, grounding, tools, latency, languages, availability). Estimate load: about 6M messages a day, roughly 70 average and 200 peak QPS, tokens per message, peak tokens per second and monthly cost per model option. Propose: clients streaming over SSE; an API layer with auth and rate limits; a stateless chat service that loads history from Postgres, assembles context within a budget and retrieves articles from a vector index; a gateway with routing, fallbacks and caching; input and output guardrails; background jobs for summaries and analytics; tracing, cost dashboards and online evals. Discuss trade-offs and cost levers.

How do you keep the chatbot fast?

Model answer

Stream responses, keep TTFT low with prompt caching and shorter context, route simple questions to smaller, faster models, cache common answers, place services near users, run guardrails in parallel, and set timeouts with fallbacks. Track p50 and p95 TTFT per route.

How do you handle conversation memory at this scale?

Model answer

Store every message in a database keyed by conversation and user. On each turn, load recent turns plus a running summary (updated asynchronously when the conversation grows) and any pinned facts, within a token budget. Keep services stateless so any instance can handle any turn; use Redis for short-lived session data.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

QPS (queries per second): How many requests a system receives each second. Daily volume divided by 86,400 gives the average; peaks are usually a few times higher.
Time to first token (TTFT): How long a user waits from sending a request until the first token of the reply arrives. Mostly set by queueing and prompt processing (prefill); it's the latency users feel most.
Model routing: Sending each request to the most suitable model, typically cheap models for easy requests and strong models for hard ones.
Prompt caching: A provider feature that reuses the processed beginning of a prompt when it repeats exactly, making those tokens much cheaper and faster.
Guardrail: A check on an AI system's input or output (rules, classifiers or a model call) that can block, modify or flag content.
Observability: Being able to understand what a system is doing from its outputs (logs, metrics and traces), so you can find out why a request was slow, wrong or expensive.

Before and after

Read first