Area 04 // EvalsLesson 76 / 14013 min read

P95 latency and cost per request

P95 latency is the response time that 95% of requests beat, which shows what your slowest real users experience far better than an average, and cost per request adds up every token, model call, tool and retry behind one user request so you know what the feature really costs.

On this page

Think of it like this
The problem it solves
How it works: P95 latency
How it works: cost per request
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A pizza shop says "our average delivery time is 25 minutes". But if one order in twenty takes 70 minutes, those customers don't care about the average; they're furious. The 95th percentile ("95% of pizzas arrive within 38 minutes") describes the experience that actually drives complaints.

And the true cost per pizza isn't just dough and cheese. It includes the box, the delivery, the occasional remake and the free pizza when something goes wrong.

The problem it solves

Teams often report "average response time 1.2 s" and "the model costs $2 per million tokens". Both hide the truth. Averages mask the slow tail that drives user frustration and timeouts. Per-token prices hide that one user request triggers several model calls (query rewriting, the main answer, a safety check, an LLM judge, retries), each with its own tokens. Decisions about models, caching and architecture need the real numbers.

How it works: P95 latency

Measure end-to-end latency per request from the user's point of view (and per step, from traces).
Sort them: the percentile pN is the value N% of requests are faster than.
Report p50, p95 and p99: p50 is the typical experience; p95 and p99 describe the tail that many users still hit every day.
For streaming chat, also track time to first token and tokens per second (streaming and latency).
Find what's in the tail: long outputs, big contexts, reasoning-heavy requests, retries after rate limits, cold caches, slow tools. Traces show which step.
Set targets (SLOs) such as "p95 under 3 s" and alert on breaches.
Compute percentiles over windows (per hour or day) and per segment (model, feature, region); never average percentiles across servers.

How it works: cost per request

List every model call a user request triggers: rewriting, retrieval embeddings, the main answer, guardrails, judges, retries, agent steps.
For each: tokens × price (input, cached input, output and reasoning tokens at their own prices) (token pricing).
Add other costs where they matter: embeddings, rerankers, vector database, tools and external APIs, GPU time if self-hosted.
Cost per request = the sum. Multiply by traffic for monthly cost.
Track it per feature, model, tenant and user: a few heavy users or features often dominate.
Better still, track cost per successful task: a cheaper request that fails and gets retried isn't cheaper.
Alert on sudden changes in cost per request or total cost.

Latency is about the tail; cost is about every call behind the request.

See it in code

Install: pip install numpy
Needs: Python 3.10+. Prices are examples as of November 2026.

How the average hides a slow tail, and what one request really costs.

latency_and_cost.py

import numpy as np
 
rng = np.random.default_rng(1)
# 10,000 request latencies: most are fast, a few are very slow (long answers, retries, cold caches).
latency_ms = np.concatenate([rng.normal(900, 150, 9_500), rng.normal(6_000, 1_500, 500)])
 
print(f"average: {latency_ms.mean():6.0f} ms   <- hides the slow tail")
for p in (50, 95, 99):
    print(f"p{p}    : {np.percentile(latency_ms, p):6.0f} ms")
 
# Cost per request: tokens x price, plus everything else the request triggered.
PRICE_IN, PRICE_OUT = 2.00, 10.00                 # $ per 1M tokens (example, Nov 2026)
calls = [  # one user request = several model calls (rewrite, answer, judge)
    {"step": "rewrite query", "in": 300,   "out": 30},
    {"step": "answer",        "in": 4_200, "out": 350},
    {"step": "safety check",  "in": 600,   "out": 5},
]
cost = sum(c["in"] / 1e6 * PRICE_IN + c["out"] / 1e6 * PRICE_OUT for c in calls)
print(f"cost per request: ${cost:.4f}  -> ${cost * 1_000_000:,.0f} per million requests")

Output

average:   1152 ms   <- hides the slow tail
p50    :    908 ms
p95    :   1464 ms
p99    :   7209 ms
cost per request: $0.0140  -> $14,050 per million requests

The average (1.15 s) looks fine, but 1% of users wait over 7 seconds. And the "answer" call is only part of the cost: the rewrite and safety check add about 13% on top.

A real example

Example

The feature that cost three times its estimate

A document-summary feature was estimated at $0.004 per request from the main model call. Its cost dashboard showed $0.013. Breaking down cost per request by step: the main call $0.004; a judge call scoring every summary $0.005 (using an expensive model); retries after timeouts on long documents $0.002; embeddings for related-document suggestions $0.002.

They moved the judge to a cheaper calibrated model on a 10% sample, raised timeouts to reduce retries, and cached embeddings. Cost per request fell to $0.005. The same breakdown showed p99 latency was dominated by the retries, which also improved.

The easy way

Start here

From your traces, compute three numbers for each feature: p50 and p95 latency, and cost per request (all model calls added up). Put them on a dashboard next to request volume. Look at them every week.

When you grow: the production setup

Teams define latency SLOs per feature (p95, p99, TTFT), compute percentiles correctly from histograms, break down latency and cost per step from traces, track cost per successful task and per tenant, set budgets and anomaly alerts, and use the numbers to drive routing, caching and model choices (cost optimization, throughput vs latency).

Trade-offs

What you gain

Percentiles reveal the slow experiences that drive complaints.
Full cost per request exposes hidden calls and waste.
Both guide model, caching and architecture decisions with real numbers.

What it costs you

Needs complete tracing of every call and step.
Percentiles need enough data and careful aggregation.
Cost attribution for shared infrastructure is approximate.

When not to use it

Never rely on averages alone for latency; even small apps should look at p95.

Common mistakes

Reporting average latency only.
Averaging percentiles from different servers or time windows.
Counting only the main model call in cost.
Ignoring reasoning tokens and retries.
Not tracking cost per tenant, so one customer quietly eats the margin.

Interview questions

Try answering out loud first, then open the model answer.

Why use p95 or p99 latency instead of the mean?

Model answer

Latency distributions for LLM apps are skewed with long tails (long outputs, large contexts, retries, cold caches). The mean is pulled by but doesn't describe the tail, while users experience individual requests. p95 and p99 show how slow the slowest common experiences are, which drive dissatisfaction, timeouts and SLO breaches. Report p50 alongside for the typical case.

How would you calculate the cost per request of a RAG chatbot?

Model answer

Sum, per request, every billed component: query rewrite and routing calls, embedding of the query, reranking, the main generation (input, cached input, output and reasoning tokens at their prices), guardrail and judge calls, retries and fallbacks, plus vector DB and tool costs and amortised infrastructure if self-hosted. Compute from traces, track per feature and tenant, and compare with cost per successful outcome.

Your p99 latency spiked while p50 stayed flat. How do you investigate?

Model answer

Filter traces in the p99 band and compare with typical ones: which step dominates (model call, tool, retrieval)? Look for retries and rate-limit errors, unusually long outputs or contexts, reasoning effort changes, specific tenants or inputs, cold starts, and provider incidents. Fix the specific slow path (timeouts, fallbacks, limits, caching) and add an alert on p99.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

P95 latency: The response time that 95% of requests are faster than; describes what the slowest common requests feel like.
Percentile: The value below which a given share of measurements fall. p95 latency means 95% of requests were faster.
Cost per request: The total cost of one user request, adding up every model call, tool and service it triggered.
Time to first token (TTFT): How long it takes from sending a request until the first token of the reply arrives. Key for perceived speed in chat.
Tail latency: The slowest part of the latency distribution (p95, p99 and above), often caused by retries, long outputs or slow steps.

Before and after

Read first