Area 00 // FoundationsLesson 07 / 14013 min read
Probability, just enough for AI
An LLM doesn't pick the next word for certain; it gives every possible word a probability, softmax turns its raw scores into those probabilities, and sampling picks one at random following them, which is why the same prompt can give different answers.
Think of it like this
Analogy
A weather forecaster says "70% chance of rain tomorrow". They're not certain; they're saying that on 10 days like this, it rains on about 7. Now imagine a spinner divided into slices: a big 70% slice labelled "rain" and a 30% slice labelled "dry". Spin it, and it usually lands on rain, but not always.
An LLM does this for every word it writes. It builds a spinner with one slice per possible next word, sized by how likely that word is, and spins.
The problem it solves
If a language model always chose its single most likely next word, every answer to "write a poem about the sea" would be identical, and it would often fall into repetitive loops ("the sea, the sea, the sea…"). If it chose words completely at random, it would write nonsense.
What we need is a way to say "usually pick a likely word, sometimes a less likely one, and never a silly one". That's probability, softmax and sampling, and it explains settings you'll meet everywhere, like temperature and top-p.
How it works, step by step
- A probability is a number from 0 to 1. 0 = never, 1 = certain, 0.25 = one time in four.
- A probability distribution lists every possible outcome with its probability, and they add up to 1. "Next word: cat 0.59, dog 0.36, pizza 0.05…"
- A model outputs logits first. For every word in its vocabulary it gives a raw score: cat 3.0, dog 2.5, pizza 0.5, banana −1.0. Scores can be any size and don't add up to anything.
- Softmax turns logits into probabilities. It raises e (about 2.718) to the power of each score, which makes them all positive and widens the gaps, then divides each by the total so they add up to 1.
- Temperature reshapes the distribution. Before softmax, divide every logit by the temperature T. T below 1 makes the top words even more dominant (focused, predictable). T above 1 flattens the spinner (varied, creative, riskier).
- Sampling spins the wheel. Pick one word at random, following the probabilities. Then the chosen word is added to the text, and the model does it all again for the next word.
You'll go deeper into decoding strategies (greedy, top-k, top-p) in lesson 17. Everything there builds on these six steps.
See it in code
- Install
pip install numpy- Needs
- Python 3.10+
import numpy as np
words = ["cat", "dog", "pizza", "banana"]
logits = np.array([3.0, 2.5, 0.5, -1.0]) # raw scores a model gave each next word
def softmax(z, temperature=1.0):
z = z / temperature # temperature reshapes the scores first
e = np.exp(z - z.max()) # subtract the max for numerical safety
return e / e.sum() # now they're positive and add up to 1
for t in (1.0, 0.3, 2.0):
probs = softmax(logits, t)
print(f"T={t}:", {w: round(float(p), 2) for w, p in zip(words, probs)})
# Sampling: pick a word at random, but following the probabilities.
rng = np.random.default_rng(42)
picks = rng.choice(words, size=1000, p=softmax(logits))
print("1000 samples at T=1:", {w: int((picks == w).sum()) for w in words})T=1.0: {'cat': 0.59, 'dog': 0.36, 'pizza': 0.05, 'banana': 0.01}
T=0.3: {'cat': 0.84, 'dog': 0.16, 'pizza': 0.0, 'banana': 0.0}
T=2.0: {'cat': 0.45, 'dog': 0.35, 'pizza': 0.13, 'banana': 0.06}
1000 samples at T=1: {'cat': 582, 'dog': 364, 'pizza': 43, 'banana': 11}At T=0.3 "pizza" and "banana" almost vanish. At T=2.0 they get real chances. And 1,000 samples land close to the 59/36/5/1 split, just as the spinner predicts.
A real example
Example
Why a support bot and a story writer use different settings
A bank's support bot answers "what's my card limit?". You want the same, careful answer every time, so the team uses a low temperature (or a model setting that makes output predictable). A children's story app wants fresh stories each night, so it samples with more randomness.
Note: as of November 2026 some providers no longer let you set temperature on their newest models at all (Anthropic's Claude API rejects temperature, top_p and top_k from API version 2026-10-21). For those models you steer variety with the prompt and other settings instead, which lesson 17 covers.
The easy way
Start here
Remember one sentence: the model gives probabilities, and sampling rolls the dice. Run the snippet, change the temperatures, and watch the distribution sharpen and flatten.
When you grow: the production setup
In production, randomness is something you design for: you run evals several times to measure variance, use structured outputs where answers must be exact, and choose (or fix) decoding settings per use case. Log the settings with every request so you can reproduce behaviour.
Trade-offs
What you gain
- Sampling gives natural, varied text instead of rigid, repetitive output.
- Temperature-like settings give a simple dial between predictable and creative.
- Probabilities tell you how confident the model is about each token, which some apps use to flag uncertainty.
What it costs you
- The same input can give different outputs, which makes testing harder.
- A low-probability bad choice can still be sampled now and then.
- High-probability doesn't mean true: a model can be confidently wrong.
When not to use it
- You need exactly the same output every time (an ID, a calculation): don't rely on sampling; compute it in code or use structured outputs and validation.
Common mistakes
- Treating a model's probability as the chance an answer is correct. It's how likely the token is given the text, not a fact check.
- Thinking temperature 0 guarantees identical outputs. Hardware and batching can still cause tiny differences.
- Turning temperature up to "make the model smarter". It makes it more random, not smarter.
- Expecting every provider to accept the same sampling settings; check each model's docs.
Interview questions
Try answering out loud first, then open the model answer.
What is softmax and why is it used at the output of a language model?
Model answer
Softmax maps a vector of real-valued logits to a probability distribution: exponentiate each, divide by the sum. It keeps the ordering of scores, makes everything positive and summing to 1, and amplifies differences. A language model uses it to turn its score for every vocabulary token into a next-token distribution that can be sampled or used in the cross-entropy loss during training.
How does temperature change model output?
Model answer
Temperature divides the logits before softmax. T < 1 sharpens the distribution so high-probability tokens dominate, giving focused, more deterministic text; T → 0 approaches greedy decoding. T > 1 flattens it, raising the chance of unusual tokens, which increases diversity but also errors and incoherence. It doesn't change what the model knows, only how it chooses.
A stakeholder complains the chatbot gives different answers to the same question. How do you respond?
Model answer
Explain that generation samples from a probability distribution, so variation is expected. Then decide whether it's a problem for this use case. If answers must be consistent, reduce randomness where the model allows it, ground answers in retrieved data, use structured outputs for facts, cache answers to common questions, and measure consistency in evals by running each test several times.
Quiz
Pick an answer to see whether it’s right and why.
0 of 6 answered
Key terms
- Probability
- A number from 0 (never) to 1 (certain) saying how likely something is.
- Probability distribution
- A list of every possible outcome with its probability, all adding up to 1. An LLM produces one over its whole vocabulary for every next token.
- Logits
- The raw scores a model outputs for every possible next token, before softmax turns them into probabilities.
- Softmax
- A function that turns a list of raw scores into positive probabilities that add up to 1, giving bigger scores bigger shares.
- Sampling
- Picking the next token at random, following the model's probabilities, instead of always taking the most likely one.
- Temperature
- A setting that divides the logits before softmax. Low values make output focused and predictable, high values make it varied and riskier.
Before and after
Progress is saved in this browser only.