Skip to content

All lessons

Area 05 // AI System DesignLesson 86 / 14015 min read

Provider rate limits and multi-tenant design

Model providers cap how many requests and tokens you can send per minute, so your system must spread that shared capacity across all your customers fairly (multi-tenancy), with per-tenant quotas, queues and isolation so one big customer can't slow down or see data from the others.

Think of it like this

Analogy

An apartment building has one water main with a fixed flow. If one flat runs every tap and the bath at once, everyone else's shower drops to a trickle: a noisy neighbour. So the building gives each flat a fair share and a meter, and keeps each flat's water separate, so nobody gets someone else's bathwater. The water company also caps the building's total flow: that's the provider rate limit.

The problem it solves

Your SaaS product serves hundreds of customers (tenants), but all their requests go through your one account with the model provider, which allows only so many requests and tokens per minute. When a big customer runs a bulk import, small customers start getting errors and slow answers. Meanwhile, shared caches, indexes and memory risk returning one customer's data to another. You need fairness, isolation and capacity planning.

How it works: provider rate limits

  1. Providers enforce limits per account (and often per model): requests per minute, input tokens per minute, output tokens per minute, sometimes per day; higher usage tiers raise them (as of November 2026 all major providers work this way; check your console's limits page).
  2. Exceeding a limit returns 429 with a retry-after hint; overloads may return 529/503.
  3. Handle it: retry with exponential backoff and jitter, respecting retry-after; SDKs retry a few times automatically (retries).
  4. Prevent it: count your own usage with a client-side token bucket per limit, and queue requests rather than firing them all at once.
  5. Raise capacity: request higher tiers, spread load across keys, regions or providers via a gateway, use prompt caching (cached tokens may count differently), and move bulk work to batch APIs, which have separate limits.

How it works: multi-tenant design

  1. Know your tenants: every request carries a tenant ID (from authentication, never from user input).
  2. Allocate capacity: give each tenant a quota (requests and tokens per minute or per day), based on plan tier, from your total provider capacity.
  3. Enforce with token buckets per tenant (rate limiting); over-quota requests get queued, slowed or rejected with a clear message.
  4. Fair scheduling: a shared queue with per-tenant fairness or priority tiers, so one burst can't starve everyone (the noisy neighbour problem).
  5. Isolate data: retrieval filters by tenant (access control in RAG), caches and memory keyed by tenant, separate indexes or row-level security for sensitive customers.
  6. Isolate cost: track tokens and cost per tenant for billing, budgets and alerts.
  7. Offer tiers: dedicated capacity or separate provider keys for enterprise tenants; bring-your-own-key options.
  8. Monitor per tenant: usage, 429s, latency, errors.
Many tenants, one provider limit: allocate fairly and keep data apart.

See it in code

Needs
Nothing: plain Python 3.

Splitting a 60,000 tokens-per-minute provider limit between three tenants with token buckets. Over-quota requests get a 429 from you before the provider rejects anyone.

tenant_quotas.py
class TokenBucket:
    """Refills `rate` units per second up to `capacity`. Used for requests/min AND tokens/min."""
    def __init__(self, capacity, rate):
        self.capacity, self.rate, self.level, self.t = capacity, rate, capacity, 0.0
    def take(self, amount, now):
        self.level = min(self.capacity, self.level + (now - self.t) * self.rate)
        self.t = now
        if amount <= self.level:
            self.level -= amount
            return True
        return False
 
# Our provider limit: 60,000 input tokens per minute. Split it fairly between tenants.
TENANT_TPM = {"acme": 30_000, "globex": 20_000, "free-tier": 10_000}
buckets = {t: TokenBucket(cap, cap / 60) for t, cap in TENANT_TPM.items()}
 
traffic = [(0, "acme", 12_000), (1, "globex", 8_000), (2, "free-tier", 9_000),
           (3, "free-tier", 9_000), (4, "acme", 12_000), (5, "acme", 12_000)]
for second, tenant, tok in traffic:
    ok = buckets[tenant].take(tok, now=second)
    print(f"t={second}s {tenant:9} {tok:6,} tokens -> {'sent' if ok else '429: queue or retry later'}")
Output
t=0s acme      12,000 tokens -> sent
t=1s globex     8,000 tokens -> sent
t=2s free-tier  9,000 tokens -> sent
t=3s free-tier  9,000 tokens -> 429: queue or retry later
t=4s acme      12,000 tokens -> sent
t=5s acme      12,000 tokens -> 429: queue or retry later

The free tier's second big request and Acme's third are held back, so Globex is never starved. In production these buckets live in Redis (shared across servers) and the rate-limiter playground shows the same algorithms live.

A real example

Example

The import that took everyone down

A B2B writing assistant ran everything through one provider key. One Monday a large customer imported 50,000 documents for AI summaries; the import hit the provider's token limit within minutes, and every other customer saw 429 errors for an hour.

The fix: per-tenant token buckets in a gateway, bulk operations moved to a separate async pipeline using the batch API, a fair queue for interactive requests, and enterprise customers given their own provider keys. The next big import only slowed that customer's own bulk job.

The easy way

Start here

Add a tenant ID to every request, keep a per-tenant counter of tokens per minute (in Redis or memory for one server), and reject or queue requests over the limit with a friendly message. Send bulk jobs to a background queue, never through the interactive path.

When you grow: the production setup

Teams run a gateway with distributed per-tenant RPM/TPM buckets, priority tiers and fair queuing, capacity planning against provider tiers, multiple keys and providers, separate paths for interactive and bulk traffic, tenant-scoped data at every layer, per-tenant cost tracking and billing, and dashboards plus alerts per tenant.

Trade-offs

What you gain

  • Fair, predictable service for every customer.
  • No surprise outages from one customer's burst.
  • Per-tenant cost tracking enables pricing and budgets.

What it costs you

  • More infrastructure (distributed limiters, queues, gateway).
  • Strict quotas can frustrate legitimate heavy users.
  • Capacity planning against changing provider limits.

When not to use it

  • A single-tenant internal tool with low traffic: simple retries with backoff may be enough.

Common mistakes

  • Sending all tenants' traffic through one unmanaged key and queue.
  • Retrying 429s immediately in a tight loop.
  • Counting only requests, not tokens.
  • Caches or indexes not scoped by tenant.
  • Taking the tenant ID from request parameters instead of authentication.

Interview questions

Try answering out loud first, then open the model answer.

How do you handle provider rate limits in an LLM application?

Model answer

Know the RPM and TPM limits per model and tier; track usage client-side with token buckets and queue requests rather than bursting; on 429s, retry with exponential backoff and jitter respecting retry-after; separate bulk work into batch APIs; raise capacity with higher tiers, multiple keys or regions and fallback providers behind a gateway; use caching to reduce token usage; and monitor 429 rates.

Design multi-tenancy for an AI SaaS product.

Model answer

Authenticate requests to a tenant; enforce per-tenant quotas (requests and tokens) with distributed token buckets and fair or priority queuing; separate interactive and bulk paths; isolate data at every layer (tenant-filtered retrieval, tenant-keyed caches and memory, row-level security, optionally dedicated indexes or keys for enterprise tenants); track cost per tenant; monitor and alert per tenant; and test for cross-tenant leakage.

What is the noisy neighbour problem and how do you solve it?

Model answer

When tenants share capacity, one tenant's heavy usage consumes resources others need, degrading their latency or causing errors. Solve it with per-tenant limits and quotas, fair scheduling or weighted queues, separate pools or keys for large tenants, isolating bulk workloads, and autoscaling where possible, monitored per tenant.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01What does an HTTP 429 from a model provider mean?
  2. 02Why are tokens-per-minute limits often hit before requests-per-minute?
  3. 03What is a noisy neighbour in a multi-tenant system?
  4. 04Which must be scoped by tenant?
  5. 05How can you raise your total capacity beyond one provider limit?

0 of 5 answered

Key terms

Rate limit
A cap on how many requests or tokens a client may send in a period. Providers return HTTP 429 when you exceed theirs.
Tokens per minute (TPM)
A provider rate limit on how many input or output tokens an account can use each minute; with long prompts it is often reached before the request limit.
Multi-tenancy
One system serving many customers (tenants) on shared infrastructure, while keeping their data, usage and costs separate.
Noisy neighbour
A tenant whose heavy usage slows down or breaks service for other tenants sharing the same capacity.
Token bucket
A rate-limiting algorithm where a bucket refills at a steady rate and each request takes some units out; when it's empty, requests wait or are rejected. Allows short bursts.
Tenant isolation
Keeping each customer's data, caches, memory and limits separate so one tenant can never see or affect another's.

Before and after

Read first

Read next

Progress is saved in this browser only.