Area 01 // LLMsLesson 19 / 14015 min read

Pretraining and post-training (SFT, RLHF, DPO)

An LLM is made in two stages, pretraining on huge amounts of raw text so it learns language and facts by predicting the next token, then post-training (supervised fine-tuning and preference training like RLHF or DPO) so it follows instructions, is helpful and refuses harmful requests.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Training an LLM is like raising a professional.

Pretraining = reading the whole library. For years, the student reads everything: books, websites, code, forums. They pick up language, facts and how people think. But ask them a question and they might just recite more of whatever book it reminds them of.
SFT = apprenticeship. A mentor shows thousands of examples: "when a customer asks this, a good answer looks like this."
RLHF / DPO = feedback from customers. For each situation, people say which of two answers they prefer, and the professional gradually leans towards the preferred style: helpful, honest, safe.

The problem it solves

Next-token prediction on internet text produces a model that knows a lot but behaves like the internet: it continues text, rambles, imitates bad examples and has no idea it's supposed to be an assistant. Companies need models that follow instructions, answer helpfully, admit uncertainty and refuse harmful requests.

Knowing how that behaviour is created explains things you'll see daily: why chat models differ from base models, why models sometimes refuse, why they're agreeable, and what fine-tuning can and can't fix.

How it works, step by step

Pretraining. Collect trillions of tokens (web pages, books, code, papers), clean and deduplicate them, then train a transformer to predict the next token everywhere (gradient descent at enormous scale, weeks on thousands of GPUs). Output: a base model.
Supervised fine-tuning (SFT). Train further on tens or hundreds of thousands of example conversations written or checked by people (and increasingly generated by stronger models): instruction → ideal response. The model learns the assistant format and style.
Preference training. Make the model prefer better answers:
- RLHF: for a prompt, sample several answers; humans rank them; train a reward model to predict those rankings; then use reinforcement learning to update the LLM to score higher on the reward model (while staying close to the SFT model so it doesn't drift into nonsense).
- DPO (direct preference optimisation): skip the reward model; train directly on pairs of (chosen, rejected) answers with a loss that raises the probability of the chosen one relative to the rejected one. Simpler, cheaper, popular for open models.
- AI feedback: labs also use models to judge answers against written principles, reducing human labelling.
Reinforcement learning on verifiable tasks. For reasoning models, RL rewards correct final answers on maths problems, passing unit tests in code, and so on, which teaches long step-by-step thinking.
Safety training and evaluation run throughout: red-teaming, refusal training, and evals before release. Together, steps 2–5 are called post-training.

From raw text to a helpful assistant.

Stage	Data	Teaches	Cost
Pretraining	Trillions of raw tokens	Language, knowledge, patterns	Millions of dollars and up
SFT	10K–1M curated conversations	Following instructions, format, tone	Small fraction of pretraining
RLHF / DPO	Ranked answer pairs	Preferences: helpful, honest, harmless	Small fraction
RL on verifiable tasks	Problems with checkable answers	Multi-step reasoning	Growing share of total compute

See it in code

Needs: Nothing: plain Python 3.

Pretraining in miniature (learn which word follows which from raw text, then generate), followed by what real post-training data looks like.

two_stages.py

import json, random
from collections import Counter, defaultdict
 
# 1) "Pretraining" in miniature: learn which word follows which from raw text.
corpus = ("the cat sat on the mat . the dog sat on the rug . "
          "the cat ate the fish . the dog ate the bone .").split()
follows = defaultdict(Counter)
for a, b in zip(corpus, corpus[1:]):
    follows[a][b] += 1                      # self-supervised: the label is just the next word
 
random.seed(0)
word, out = "the", ["the"]
for _ in range(6):
    choices = follows[word]
    word = random.choices(list(choices), weights=list(choices.values()))[0]
    out.append(word)
print("base model writes:", " ".join(out))
 
# 2) Post-training data formats (what SFT and DPO datasets look like).
sft_example = {"messages": [
    {"role": "user", "content": "What does a cat eat?"},
    {"role": "assistant", "content": "Cats mostly eat meat, such as fish or cat food."}]}
dpo_example = {"prompt": "What does a cat eat?",
               "chosen": "Cats mostly eat meat, such as fish or cat food.",
               "rejected": "the cat ate the fish . the dog"}
print("SFT row :", json.dumps(sft_example)[:60], "...")
print("DPO keys:", list(dpo_example))

Output

base model writes: the fish . the mat . the
SFT row : {"messages": [{"role": "user", "content": "What does a cat e ...
DPO keys: ['prompt', 'chosen', 'rejected']

The "base model" produces fluent-ish text but doesn't answer anything; that's the gap post-training fills. SFT datasets are lists of conversations in a messages format; preference datasets are (prompt, chosen, rejected) triples. You'll use exactly these formats if you fine-tune a model.

A real example

Example

Why one model refuses and another doesn't

A company tests two open models from Hugging Face for an internal tool. The "base" version of a model, asked "Summarise this email:", writes a fake email signature and three more emails. The "instruct" version of the same model writes a neat summary. Same pretraining, different post-training.

Later, the instruct model refuses a legitimate security-testing request because its safety training was cautious. The team didn't retrain it; they rewrote the system prompt to give context about the authorised engagement, which resolved most refusals. Knowing where behaviour comes from tells you which lever to pull.

The easy way

Start here

Know the two labels you'll see on model hubs: base (pretrained only, continues text) and instruct / chat (post-trained, follows instructions). For building apps, pick instruct/chat models. Leave pretraining to the labs.

When you grow: the production setup

When prompting isn't enough, teams fine-tune: SFT on a few thousand high-quality examples from their own domain, sometimes followed by DPO on pairs where users preferred one answer. They hold out an eval set, compare against the base instruct model, and watch for regressions in general ability and safety (LoRA, RAG vs fine-tuning).

Trade-offs

What you gain

Pretraining gives broad knowledge and skills from unlabelled data.
Post-training turns that knowledge into a useful, steerable, safer assistant with comparatively little data.
DPO and AI feedback make preference training cheaper and accessible to smaller teams.

What it costs you

Pretraining is enormously expensive and its data contains errors and biases.
Post-training can make models overly cautious, flattering ("sycophantic") or verbose.
Knowledge is frozen at the training cut-off date.

When not to use it

Don't fine-tune to add frequently changing facts; use retrieval. Fine-tune for behaviour, format and style.

Common mistakes

Using a base model in a chat app and wondering why it rambles instead of answering.
Expecting fine-tuning to teach lots of new facts reliably (it's better at style and format).
Assuming a model knows about events after its training cut-off.
Treating refusals or flattery as random; they're learned behaviours from post-training you can often steer with the prompt.

Interview questions

Try answering out loud first, then open the model answer.

Explain the stages of training a chat LLM.

Model answer

Pretraining: self-supervised next-token prediction on trillions of tokens, producing a base model with broad knowledge. Supervised fine-tuning on curated instruction-response pairs teaches instruction following and the assistant format. Preference optimisation (RLHF with a reward model and an RL algorithm like PPO, or DPO directly on preference pairs) aligns outputs with human preferences for helpfulness and harmlessness. Many modern models add RL on verifiable tasks to improve reasoning, plus safety training and extensive evals.

What's the difference between RLHF and DPO?

Model answer

RLHF trains a separate reward model on human preference comparisons, then optimises the policy with reinforcement learning (e.g. PPO) against that reward, with a KL penalty to stay close to the reference model. It's powerful but complex and can be unstable. DPO derives a closed-form objective that optimises the policy directly from chosen/rejected pairs, implicitly encoding the reward, with no reward model or RL loop. It's simpler, cheaper and more stable, though RL-based methods remain useful especially with verifiable rewards.

When would you fine-tune a model rather than prompt it?

Model answer

When prompting and retrieval can't reliably get the needed behaviour: a consistent output format or tone, a specialised task where examples beat instructions, reducing prompt length and latency at high volume, or distilling a large model's behaviour into a smaller cheaper one. Not for injecting fast-changing knowledge, which belongs in retrieval. Always measure with a held-out eval against the prompted baseline.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Pretraining: The first, most expensive training stage, where a model learns by predicting the next token on huge amounts of raw text.
Post-training: Training after pretraining (SFT, preference training, RL, safety) that turns a base model into a helpful assistant.
Base model: A model after pretraining only. It continues text rather than following instructions.
Supervised fine-tuning (SFT): Training a model on examples of prompts paired with ideal responses, so it learns to follow instructions in a given style.
RLHF: Reinforcement learning from human feedback. Humans rank answers, a reward model learns their preferences, and RL pushes the model towards preferred answers.
DPO: Direct preference optimisation. Trains a model directly on pairs of preferred and rejected answers, without a separate reward model or RL loop.
Reward model: A model trained to predict how much humans would prefer an answer; used to guide reinforcement learning.

Before and after

Read first