Area 05 // AI System DesignLesson 87 / 14013 min read

RAG vs fine-tuning

RAG gives a model the right knowledge at question time (fresh, citable, permission-aware), while fine-tuning changes how the model behaves (style, format, narrow skills), so most systems use RAG for knowledge, fine-tuning only for behaviour, and often both together.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Hiring a new nurse: you can train them (fine-tuning) so they write notes in the hospital's format and follow its procedures automatically. But you don't make them memorise every patient's file; they look it up in the records system each time (RAG), because files change daily, need to be cited in reports, and some are restricted.

Training shapes how they work; lookups give them what they need to know right now.

The problem it solves

"Should we fine-tune or use RAG?" is one of the most common AI design questions, and it's often answered wrongly: teams fine-tune a model on their documents hoping it will "know" them, then find it still hallucinates, can't cite, leaks restricted information and goes out of date. Others build elaborate RAG to fix what is really a tone or format problem. The two tools solve different problems.

How it works, step by step

Start with prompting and examples (prompt engineering, few-shot). Build an eval set first.
Diagnose the gap with error analysis:
- The model doesn't know something (private, recent, detailed facts) → a knowledge gap.
- The model knows enough but behaves wrongly (format, tone, following a procedure, a narrow skill) → a behaviour gap.
Knowledge gap → RAG (or tools for live data):
- Fresh: update a document and it's used immediately.
- Citable: answers point to sources (citations).
- Permission-aware: filter by who's asking (access control).
- Cost: retrieval infrastructure and longer prompts.
Behaviour gap → fine-tuning (LoRA):
- Consistent format, tone and procedures without long prompts.
- A smaller, cheaper model matching a bigger one on a narrow task (distillation).
- Cost: training data, training runs, versioning, re-training when base models change.
Both: a fine-tuned model that's good at your format and domain language, fed current facts by RAG. Common in mature systems.
Decide with evals: compare prompted, RAG, fine-tuned and combined variants on the same golden set, along with cost and latency.

Need	RAG	Fine-tuning
Fresh / changing facts	Yes	No (frozen)
Citations	Yes	No
Per-user permissions	Yes	No
Consistent format and tone	Partly (via prompt)	Yes
Narrow skill / cheaper model	No	Yes
Shorter prompts at high volume	No (adds context)	Yes

Pick the tool by the kind of gap.

See it in code

Needs: Nothing: plain Python 3. A rule-of-thumb helper; evals make the real decision.

Turning the requirements of two features into advice.

rag_or_finetune.py

def rag_or_finetune(need: dict) -> list[str]:
    """A rule-of-thumb helper. Real decisions come from evals on both options."""
    advice = []
    if need["knowledge_changes_often"] or need["needs_citations"] or need["per_user_permissions"]:
        advice.append("RAG: knowledge that changes, must be cited or depends on who is asking")
    if need["consistent_format_or_tone"] or need["narrow_task_high_volume"]:
        advice.append("fine-tune: behaviour, format or a narrow skill at high volume")
    if need["prompt_too_long_and_costly"]:
        advice.append("fine-tune (or distil) to move long instructions into the weights")
    return advice or ["start with prompting; measure before adding either"]
 
support_bot = dict(knowledge_changes_often=True, needs_citations=True, per_user_permissions=False,
                   consistent_format_or_tone=True, narrow_task_high_volume=False,
                   prompt_too_long_and_costly=False)
ticket_tagger = dict(knowledge_changes_often=False, needs_citations=False, per_user_permissions=False,
                     consistent_format_or_tone=True, narrow_task_high_volume=True,
                     prompt_too_long_and_costly=True)
for name, need in [("support bot", support_bot), ("ticket tagger", ticket_tagger)]:
    print(name, "->")
    for a in rag_or_finetune(need):
        print("   ", a)

Output

support bot ->
    RAG: knowledge that changes, must be cited or depends on who is asking
    fine-tune: behaviour, format or a narrow skill at high volume
ticket tagger ->
    fine-tune: behaviour, format or a narrow skill at high volume
    fine-tune (or distil) to move long instructions into the weights

The support bot needs RAG for its knowledge and may benefit from fine-tuning for its reply format (both); the ticket tagger is a pure behaviour problem at high volume, a classic fine-tuning case.

A real example

Example

The insurer that fine-tuned on its policies, then switched to RAG

An insurer fine-tuned a model on 3,000 pages of policy documents so it would "know the policies". In testing it answered confidently but mixed up clauses between products, couldn't say where an answer came from, and every policy update meant another training run. Some clauses were confidential to certain partners, yet the model would mention them to anyone.

They moved policy knowledge to RAG with citations and partner-level permission filters, and kept a small fine-tune only for the claim-letter format their compliance team required. Accuracy on their eval set rose from 71% to 92%, and policy updates went live the same day.

The easy way

Start here

Ask one question: "Is the model missing information, or doing the wrong thing with information it has?" Missing information → RAG. Wrong behaviour after good prompting → fine-tuning. Measure both against the same eval set before committing.

When you grow: the production setup

Teams treat it as an empirical decision: baseline prompting, RAG variants, fine-tuned variants and combinations evaluated on the same golden set with cost and latency; fine-tuned models versioned in a registry and re-evaluated when base models update; RAG pipelines monitored for freshness; and distillation used to cut cost once a pipeline works.

Trade-offs

What you gain

RAG: fresh, citable, permission-aware knowledge without training.
Fine-tuning: consistent behaviour, shorter prompts, cheaper smaller models.
Combined: the strengths of both.

What it costs you

RAG: retrieval infrastructure, longer prompts, quality depends on retrieval.
Fine-tuning: data collection, training, versioning, staleness, possible forgetting.
Combined: two systems to maintain and evaluate.

When not to use it

Avoid fine-tuning to inject changing or permissioned facts.
Avoid RAG for pure style and format problems.

Common mistakes

Fine-tuning on documents to make a model "know" them.
Building complex RAG to fix tone or format.
Skipping prompting and few-shot as a baseline.
Comparing approaches without a shared eval set.
Forgetting that fine-tunes must be redone when the base model is retired.

Interview questions

Try answering out loud first, then open the model answer.

RAG or fine-tuning: how do you decide?

Model answer

Diagnose whether the gap is knowledge or behaviour. Knowledge that is private, changing, needs citations or depends on permissions belongs in RAG (or tools). Behaviour such as format, tone, following domain procedures, a narrow task, or shrinking cost and prompt length calls for fine-tuning after prompting falls short. Often combine them. Validate every option on the same eval set with cost and latency.

Why not fine-tune a model on your company documents so it knows them?

Model answer

Fine-tuning is unreliable for memorising many specific facts and can increase confident hallucinations; the knowledge is frozen and needs retraining on every change; answers can't cite sources; per-user access control is impossible because anything learned can surface for anyone; and it's costly to maintain. RAG addresses all of these.

When would you combine RAG and fine-tuning?

Model answer

When you need both current, citable knowledge and specialised behaviour: e.g. a fine-tuned model that reliably produces a regulated report format or uses domain terminology correctly, while retrieval supplies the specific facts. Fine-tuning can also teach the model to use retrieved context better (citing, abstaining) for your domain.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

RAG (retrieval-augmented generation): Fetching relevant documents for a question and adding them to the prompt, so the model answers from your data.
Fine-tuning: Further training an existing model on your own examples to change its behaviour, style or narrow skills.
LoRA: Low-rank adaptation. Fine-tuning that freezes the original weights and trains small add-on matrices, usually under 1% of the parameters.
Distillation: Training a small student model to imitate a large teacher model's outputs, keeping much of its quality at lower cost.
Eval: A repeatable test of an AI system's quality on a set of examples with a scoring method.

Before and after

Read first