Area 01 // LLMsLesson 17 / 14015 min read

Logits, decoding, temperature and top-p

At every step an LLM outputs a score (logit) for every possible next token, and a decoding strategy (greedy, beam, top-k, top-p, with a temperature) decides which token actually gets written, which controls how predictable or creative the output is.

On this page

Think of it like this
The problem it solves
How it works: from logits to the next token
How it works: temperature and top-p
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

You're finishing a friend's sentence: "For breakfast I had…". Several endings pop into your head with different likelihoods: "toast" (very likely), "eggs" (likely), "a croissant" (possible), "a bicycle" (no).

A cautious speaker always says the most likely word ("toast"): greedy.
A playful speaker sometimes picks a less obvious but sensible option: sampling with top-p, never "a bicycle".
A planner thinks a few words ahead before committing: beam search.

Decoding strategies are just these personalities, written as rules.

The problem it solves

A model doesn't output a word; it outputs a score for every one of its 100,000+ possible tokens (logits). Something has to turn those scores into one chosen token, thousands of times per answer. Choose badly and you get robotic repetition, or nonsense, or answers that change unpredictably when you need them stable.

The decoding settings are the knobs you'll see in every API and playground. Understanding them tells you which to touch and which to leave alone.

How it works: from logits to the next token

The model outputs logits: one raw score per vocabulary token for the next position.
Softmax turns logits into probabilities that sum to 1.
A decoding strategy picks one token:
- Greedy: take the highest-probability token. Deterministic, fast, but prone to repetitive, bland text and can miss better sentences.
- Beam search: keep the best B partial sequences at each step, extend each, and keep the best B again; at the end pick the sequence with the highest total probability. Common in translation and speech recognition, rarely used for chat.
- Top-k sampling: keep only the k most likely tokens, renormalise, and sample.
- Top-p (next section).
Append the chosen token and repeat until a stop token, a stop sequence or max_tokens.
Extras some APIs offer: frequency/presence penalties (discourage repeating tokens), stop sequences, logit bias (push specific tokens up or down), seeds (for more reproducible sampling).

How it works: temperature and top-p

Temperature (T) divides the logits before softmax.
- T < 1 sharpens the distribution: the top tokens dominate, output gets focused and predictable.
- T = 1 leaves the model's own distribution unchanged.
- T > 1 flattens it: unlikely tokens get real chances; output gets varied, then weird.
- T → 0 approaches greedy.
Top-p (nucleus sampling) sorts tokens by probability and keeps the smallest set whose total reaches p (say 0.9), then samples within it. When the model is confident, that set is tiny; when it's unsure, the set grows. It cuts off the long tail of nonsense tokens without fixing a number like top-k does.
Use one dial at a time. Providers usually advise changing temperature or top-p, not both.
Check what your model allows. As of November 2026, some providers restrict these settings on their newest models. Anthropic's Claude API rejects temperature, top_p and top_k on every model from API version 2026-10-21; there you steer with the prompt, structured outputs and effort settings instead. OpenAI and Google still document sampling parameters for many models, but reasoning models restrict them too. Always read the model's API page.

Use case	Typical choice
Data extraction, classification, code fixes	Low temperature (0–0.3) or the provider's default, plus structured outputs
Chat assistant	Provider default (often T ≈ 0.7–1, top-p ≈ 0.9–1)
Brainstorming, fiction	Higher temperature (≈ 1–1.2) or top-p ≈ 0.95

See it in code

Needs: Nothing: plain Python 3.

A toy "model" with fixed next-word probabilities, decoded four ways. Notice that greedy and beam search disagree.

decoding.py

# A toy "model": for a given previous word, the probabilities of the next word.
NEXT = {
    "<start>": {"the": 0.5, "a": 0.4, "my": 0.1},
    "the":   {"cat": 0.45, "dog": 0.35, "end": 0.2},
    "a":     {"dog": 0.9, "end": 0.1},
    "my":    {"cat": 0.6, "end": 0.4},
    "cat":   {"end": 1.0}, "dog": {"end": 1.0},
}
 
def greedy(word="<start>"):
    out = []
    while word != "end":
        word = max(NEXT[word], key=NEXT[word].get)  # always the single most likely
        out.append(word)
    return out
 
def top_p_choices(probs, p):
    ranked = sorted(probs.items(), key=lambda kv: -kv[1])
    keep, total = [], 0.0
    for w, pr in ranked:                           # smallest set whose sum reaches p
        keep.append(w); total += pr
        if total >= p: break
    return keep
 
def beam(width=2):
    beams = [(["<start>"], 1.0)]
    for _ in range(3):
        cand = [(s + [w], pr * p) for s, pr in beams if s[-1] != "end"
                for w, p in NEXT[s[-1]].items()] + [b for b in beams if b[0][-1] == "end"]
        beams = sorted(cand, key=lambda c: -c[1])[:width]   # keep the best few paths
    return beams[0]
 
print("greedy      :", greedy())
print("top-k (k=2) :", sorted(NEXT["<start>"], key=NEXT["<start>"].get)[-2:][::-1])
print("top-p (0.85):", top_p_choices(NEXT["<start>"], 0.85))
best, prob = beam()
print("beam (w=2)  :", best[1:], "probability", round(prob, 3))

Output

greedy      : ['the', 'cat', 'end']
top-k (k=2) : ['the', 'a']
top-p (0.85): ['the', 'a']
beam (w=2)  : ['a', 'dog', 'end'] probability 0.36

Greedy grabbed "the" (0.5) and ended with a sentence of probability 0.5 × 0.45 = 0.225. Beam search kept "a" alive and found "a dog" with 0.4 × 0.9 = 0.36: a better sentence that started with a worse word. Setting sampling parameters on a real API looks like client.responses.create(model=..., input=..., temperature=0.2) on models that accept them.

A real example

Example

One product, three decoding settings

A writing app has three features. "Fix grammar" must return the same correction each time, so it uses the most deterministic settings the model allows plus a strict output format. "Suggest titles" should feel fresh, so it samples with more randomness and asks for five options. "Summarise meeting notes" uses the provider's default sampling and is checked by evals for faithfulness.

When the team moved one feature to a model that rejects sampling parameters, they deleted the temperature setting, tightened the prompt ("give exactly one corrected version, no alternatives") and re-ran their evals to confirm nothing regressed.

The easy way

Start here

Start with the provider's defaults. Only turn down randomness (or rely on structured outputs) when you need consistency, and only turn it up for creative tasks. Try the sampling playground logic above with different numbers to build intuition.

When you grow: the production setup

Teams store decoding settings with each prompt version, run evals several times per setting to measure variance, keep per-feature defaults in config, and handle provider differences in one adapter layer (an LLM gateway), because each provider and model accepts different parameters.

Trade-offs

What you gain

Simple dials to trade consistency for creativity per feature.
Top-p adapts to the model's confidence, cutting off nonsense tokens.
Beam search can find higher-probability sequences for tasks like translation.

What it costs you

Randomness makes outputs harder to test and reproduce.
Greedy and beam search tend to produce repetitive, generic text in open-ended chat.
Settings don't transfer cleanly between models or providers, and some models don't allow them.

When not to use it

Don't use high temperature to "make answers smarter" or for factual tasks; it only adds randomness.
Don't depend on temperature 0 for exact reproducibility; use structured outputs and validation.

Common mistakes

Tuning temperature and top-p together, then not knowing which one helped.
Copying temperature=0 into code for a model that rejects the parameter.
Expecting temperature 0 to make outputs identical every time.
Using sampling settings to fix problems that are really prompt or retrieval problems.

Interview questions

Try answering out loud first, then open the model answer.

Compare greedy decoding, beam search, top-k and top-p sampling.

Model answer

Greedy picks the argmax each step: deterministic and cheap but repetitive and myopic. Beam search keeps the B best partial sequences and returns the highest-scoring complete one: better for tasks with a single correct output like translation, but bland for open-ended text. Top-k samples from the k most likely tokens, giving diversity with a fixed cutoff. Top-p samples from the smallest set covering probability mass p, adapting the cutoff to the model's confidence. Temperature reshapes the distribution before any sampling.

How would you choose decoding settings for a production feature?

Model answer

Start from the task: extraction or classification needs consistency, so use the lowest randomness available plus structured outputs; chat uses defaults; creative tasks allow more randomness. Check what the chosen model supports, because some reject sampling parameters. Then measure with evals, running each case several times to see variance, and pick the setting that maximises quality and consistency. Version the settings with the prompt.

Why do models sometimes repeat themselves and how do you fix it?

Model answer

With greedy or low-temperature decoding, once a phrase becomes likely, repeating it can stay the most likely continuation, creating a loop. Fixes: sample with moderate temperature or top-p, add frequency or presence penalties where supported, set stop sequences and sensible max tokens, and improve the prompt (clear length and format instructions).

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Logits: The raw scores a model outputs for every possible next token, before softmax turns them into probabilities.
Softmax: A function that turns a list of raw scores into positive probabilities that add up to 1, giving bigger scores bigger shares.
Greedy decoding: Always choosing the single most likely next token. Deterministic but repetitive.
Beam search: A decoding method that keeps the few best partial sequences at each step and returns the best complete one.
Top-k sampling: Sampling the next token from only the k most likely tokens.
Top-p (nucleus) sampling: Sampling from the smallest set of most likely tokens whose probabilities add up to p, such as 0.9.
Temperature: A setting that divides the logits before softmax. Low values make output focused and predictable, high values make it varied and riskier.
Sampling: Picking the next token at random, following the model's probabilities, instead of always taking the most likely one.

Before and after

Read first