Skip to content

All lessons

Area 05 // AI System DesignLesson 81 / 14013 min read

Speculative decoding and prefix caching

Speculative decoding speeds up generation by letting a small fast model guess several tokens ahead and having the big model check them all in one pass, and prefix caching skips recomputing the start of a prompt that many requests share, cutting time to first token and cost.

Think of it like this

Analogy

Speculative decoding: a junior writer drafts the next few words of a sentence, and an expert editor checks them all at once: "yes, yes, yes, no, change that one". The expert works faster checking than writing from scratch, and the final text is exactly what the expert would have written.

Prefix caching: a teacher who explains the same 20-minute introduction to every class records it once and plays the recording, only speaking live for each class's questions.

The problem it solves

Generation is slow because the big model produces one token per full forward pass, and each pass is limited by memory bandwidth (GPUs and memory). Meanwhile most requests start with the same long system prompt or documents, which the server re-processes every single time, adding cost and time to first token.

Both techniques cut that waste without changing what the model says.

How it works: speculative decoding

  1. A cheap draft proposes k tokens (e.g. 4): a small model of the same family, an n-gram lookup in the prompt (great for copying), or extra prediction heads trained on the big model (EAGLE-style).
  2. The big model verifies all k in one forward pass, which costs about the same as generating one token, because the pass is memory-bound.
  3. Accept the longest correct prefix of the guesses, plus one token the big model produces itself at the first mismatch.
  4. Repeat. If most guesses are accepted, you get several tokens per big-model pass.
  5. Quality is preserved: with exact verification (rejection sampling), the output follows the big model's own distribution.
  6. Speed-up depends on acceptance rate and draft cost: often 1.5–3× for predictable text (code, structured output, copying context), less for creative text.
  7. Available in serving engines (vLLM supports n-gram, suffix, EAGLE and other methods as of November 2026); hosted providers use similar tricks internally.

How it works: prefix caching

  1. Many requests share a prefix: the same system prompt, tool definitions, few-shot examples or document.
  2. The server stores the KV cache for that prefix the first time it's computed (KV cache).
  3. Later requests with an identical prefix reuse it: only the new tokens after the prefix need prefill.
  4. Benefits: lower time to first token, less GPU compute, more capacity.
  5. In vLLM, automatic prefix caching uses PagedAttention blocks shared between requests (vLLM).
  6. With hosted APIs, the same idea is prompt caching, which also gives a price discount.
  7. Design prompts for it: stable content first, variable content last; identical bytes matter.
Speculative decoding (left) and prefix caching (right).

See it in code

Needs
Nothing: plain Python 3. A simulation of acceptance rates and shared prefixes.

How many big-model passes speculative decoding saves at different acceptance rates, and how much prefill prefix caching skips.

speculative_and_prefix.py
import random
 
# SPECULATIVE DECODING: a small draft model guesses k tokens; the big model checks them in ONE pass.
random.seed(3)
def big_model_passes(n_tokens, k, acceptance):
    passes, made = 0, 0
    while made < n_tokens:
        accepted = 0
        while accepted < k and random.random() < acceptance:   # accept guesses until the first miss
            accepted += 1
        made += accepted + 1                                   # the big model adds one token itself
        passes += 1
    return passes
 
n = 500
print(f"normal decoding     : {n} big-model passes")
for acc in (0.5, 0.8):
    p = big_model_passes(n, k=4, acceptance=acc)
    print(f"speculative (k=4, {acc:.0%} accepted): {p} passes -> ~{n / p:.1f}x faster")
 
# PREFIX CACHING: requests that start with the same tokens reuse the stored KV cache.
system_and_docs = 3_000                                # identical start of every request
requests = [system_and_docs + q for q in (40, 25, 60, 30, 55)]
without = sum(requests)
with_cache = requests[0] + sum(r - system_and_docs for r in requests[1:])
print(f"prefill tokens computed: {without:,} without prefix cache, {with_cache:,} with it")
Output
normal decoding     : 500 big-model passes
speculative (k=4, 50% accepted): 263 passes -> ~1.9x faster
speculative (k=4, 80% accepted): 151 passes -> ~3.3x faster
prefill tokens computed: 15,210 without prefix cache, 3,210 with it

Real speed-ups are a bit lower than the pass counts suggest, because drafting and verifying k tokens cost a little extra. Prefix caching saved almost 80% of prefill work here because each question is tiny compared with the shared start.

A real example

Example

A code assistant that got 2.4× faster

A self-hosted code-completion service used a 30B model. Code is predictable (closing brackets, repeated identifiers, boilerplate), so the team enabled speculative decoding with a small draft model from the same family. Acceptance averaged about 75% and median generation time fell 2.4×, with identical outputs. They also put the repository context and instructions first in every prompt; with prefix caching, time to first token dropped from 900 ms to 250 ms for follow-up requests in the same file.

The easy way

Start here

With hosted APIs, you get the prefix-caching benefit by ordering prompts (stable first) and enabling prompt caching. When self-hosting with vLLM, turn on prefix caching (and try an n-gram speculative method, which needs no extra model) and measure tokens per second before and after.

When you grow: the production setup

Teams pick draft methods per workload (n-gram for copy-heavy RAG, EAGLE-style heads or small drafts for general text), tune k, monitor acceptance rates, watch the extra memory and compute, and combine prefix caching with cache-aware routing so requests sharing a prefix land on the same replica.

Trade-offs

What you gain

  • Speculative decoding: faster generation with unchanged output.
  • Prefix caching: big savings in time to first token and compute for shared prompts.
  • Both stack with batching and quantization.

What it costs you

  • Speculative decoding adds complexity and memory; gains vary with text type and load.
  • Prefix caching only works for identical prefixes and uses cache memory.
  • Under very heavy batching, speculative gains shrink.

When not to use it

  • Highly creative, unpredictable outputs with low acceptance (speculative decoding).
  • Every request is unique from the first token (prefix caching).

Common mistakes

  • Assuming speculative decoding lowers quality (with exact verification it doesn't).
  • Using a draft model with a different tokenizer from the target.
  • Putting variable content (timestamps, user IDs) at the start of prompts.
  • Load-balancing randomly so shared prefixes rarely hit the same cache.
  • Not measuring acceptance rate and real tokens per second.

Interview questions

Try answering out loud first, then open the model answer.

How does speculative decoding work and why doesn't it hurt quality?

Model answer

A cheap draft (small model, n-gram lookup, or auxiliary heads) proposes k tokens; the target model scores all k positions in one forward pass. Tokens are accepted left to right while they match what the target would produce (via rejection sampling for sampled decoding), and at the first rejection the target's own token is used. Since every emitted token is validated by the target's distribution, outputs are statistically identical; speed comes from verifying several tokens per memory-bound pass.

What is prefix caching and how do you design for it?

Model answer

Reusing the stored KV cache for identical prompt prefixes across requests, so only new tokens need prefill. Design prompts with stable content (system instructions, tool schemas, shared documents) first in a deterministic order and variable content last, keep traffic warm, and route requests with shared prefixes to the same replicas. With hosted APIs this appears as prompt caching with discounted pricing.

Which latency metrics do these techniques improve?

Model answer

Prefix caching mainly reduces time to first token and prefill compute (and cost). Speculative decoding mainly reduces time per output token and total generation time. Together they improve end-to-end latency, and both increase effective capacity.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01In speculative decoding, what does the small draft model do?
  2. 02Does speculative decoding change the output quality?
  3. 03When is speculative decoding most effective?
  4. 04What does prefix caching reuse?
  5. 05Which prompt layout benefits most from prefix caching?

0 of 5 answered

Key terms

Speculative decoding
Speeding up generation by letting a cheap method guess several next tokens and having the big model check them all in one pass; accepted guesses save time, and the output is unchanged.
Draft model
The small, fast model (or other guesser) that proposes tokens in speculative decoding for the large model to verify.
Prefix caching
Reusing the stored KV cache for a prompt beginning that was already processed (like a shared system prompt), so only the new part needs computing. Cuts time to first token and cost.
KV cache
Stored keys and values of already-processed tokens, so each new token can be generated without recomputing everything before it.
Prompt caching
A provider feature that reuses the processed beginning of a prompt when it repeats exactly, making those tokens much cheaper and faster.

Before and after

Read first

Read next

Progress is saved in this browser only.