Area 01 // LLMsLesson 26 / 14014 min read
Hallucinations
A hallucination is when a model states something false or made up with full confidence, because it generates likely-sounding text rather than looking facts up; you reduce it by grounding answers in real sources, allowing "I don't know", and checking outputs.
Think of it like this
Analogy
Picture a very fluent student who hates saying "I don't know". Asked about a book they never read, they'll give a confident, detailed, completely invented summary, because sounding right is what they've practised. Hand them the book and say "answer only from this, and say if it's not there", and they become reliable.
LLMs are that student. Hallucination is the confident invention. Grounding is handing them the book.
The problem it solves
Hallucinations are the number-one reason AI features fail in real use: a support bot invents a refund policy, a legal assistant cites a court case that doesn't exist, a coding assistant calls a library function that was never written. The text sounds perfect, so users believe it.
Because models are trained to produce likely text, not true text, this isn't a bug you can patch once. You need a set of habits in prompts, architecture and testing that keep it rare and catch it when it happens.
How it works, step by step
Why it happens
- Prediction, not lookup. The model picks likely next tokens (lesson 17). When it doesn't know, a plausible-sounding guess is still likely.
- Gaps in knowledge. Its training data has a cut-off date and never included your private documents.
- Pressure to answer. Post-training rewards helpful answers, so models lean towards answering over abstaining.
- Messy context. Long, noisy or contradictory context makes it more likely to mix things up.
How to reduce it
- Ground the answer. Retrieve the relevant source text and put it in the prompt, with the rule "answer only from these sources" (grounding, the core of RAG).
- Allow abstaining. "If the answer isn't in the sources, say you don't know and suggest who can help."
- Ask for citations and check them. Have the model quote or reference the source for each claim, then verify the quotes actually exist (citations).
- Use tools for facts and maths. Lookups, prices, dates and calculations belong in tool calls and code.
- Constrain outputs. Structured outputs with nullable fields stop the model from filling required fields with invented data.
- Measure it. Track faithfulness (are claims supported by the sources?) in evals (RAG evals) and watch user reports.
- Design the product for imperfection. Show sources, let users check them, keep humans in the loop where errors are costly.
See it in code
- Needs
- Nothing: plain Python 3.
A cheap, effective check: every quote the model cites must appear word for word in the source. It catches invented "facts" dressed up as citations.
import re
# A cheap hallucination check: every quote the model cites must really appear in the source.
SOURCE = """Refunds are available within 30 days of purchase.
Opened software cannot be refunded.
Gift cards are non-refundable."""
model_answer = """You can get a refund within 30 days ["Refunds are available within 30 days of purchase."].
Gift cards can be refunded at any store ["Gift cards are refundable at any store."]."""
def check_quotes(answer: str, source: str) -> list[tuple[str, bool]]:
quotes = re.findall(r'\["(.+?)"\]', answer) # pull out every ["quoted"] citation
return [(q, q in source) for q in quotes] # is it word-for-word in the source?
for quote, found in check_quotes(model_answer, SOURCE):
print("OK " if found else "FAKE", quote)
if not all(found for _, found in check_quotes(model_answer, SOURCE)):
print("-> block this answer, or regenerate it, or show it with a warning")OK Refunds are available within 30 days of purchase.
FAKE Gift cards are refundable at any store.
-> block this answer, or regenerate it, or show it with a warningIn production you'd use fuzzy matching (small wording differences) and an LLM judge for claims without direct quotes, but this one check already stops the most damaging failure: a confident, fake policy.
A real example
Example
An airline chatbot invents a refund policy
A widely reported case: an airline's website chatbot told a grieving customer they could buy a full-price ticket and claim a bereavement discount afterwards. The real policy said no such thing. The customer relied on it, the airline refused, and a tribunal ruled the airline was responsible for what its chatbot said.
The lesson for engineers: answers about policies, prices and legal terms must be grounded in the actual policy text, quote it, and refuse or hand over to a human when the policy doesn't cover the question. "The AI said it" is not a defence.
The easy way
Start here
For any factual feature, add three lines to your prompt: "Use only the sources below. Quote the source for each claim. If the sources don't contain the answer, say you don't know." Then run the quote check above on every answer.
When you grow: the production setup
Production systems combine retrieval quality work (good chunks, hybrid search, reranking), faithfulness and citation checks, structured outputs, tool use for live facts, confidence thresholds that route uncertain answers to humans, red-team tests that try to provoke invention, and online monitoring of user corrections (user feedback loops).
Trade-offs
What you gain
- Grounding and abstaining make answers far more trustworthy.
- Citations let users verify answers themselves.
- Checks catch many hallucinations automatically before users see them.
What it costs you
- Strict grounding makes the assistant say "I don't know" more often.
- Retrieval, checks and judges add latency and cost.
- No combination of techniques reaches zero hallucinations.
When not to use it
- Pure creative tasks (fiction, brainstorming) where invention is the point; heavy grounding would only get in the way.
Common mistakes
- Trusting fluent, confident answers without checking them.
- Not giving the model permission to say "I don't know".
- Treating citations as proof without verifying they exist in the source.
- Asking the model to do exact maths, counts or lookups itself.
- Measuring quality by "it looked fine in the demo" instead of faithfulness evals.
Interview questions
Try answering out loud first, then open the model answer.
What causes LLM hallucinations?
Model answer
The model is trained to generate plausible continuations, not to retrieve verified facts. Hallucinations happen when knowledge is missing or outdated, the prompt is ambiguous, the model is pushed to answer rather than abstain (a side effect of training for helpfulness), the context is long or noisy, or sampling picks a low-probability wrong token. There's no internal truth check unless the system adds one.
How would you reduce hallucinations in a customer-support bot?
Model answer
Ground answers in retrieved, up-to-date help-centre and policy content; instruct the model to answer only from sources, cite them, and say when it doesn't know; use tools for account-specific facts; constrain outputs; verify citations; route low-confidence or high-risk topics (refunds, legal) to humans; and measure faithfulness with an eval set and production sampling, iterating on retrieval quality.
How do you measure hallucination rate?
Model answer
Build a labelled eval set with source documents and questions, including unanswerable ones. Score faithfulness (are claims supported by the provided context?) using claim extraction plus an LLM judge calibrated against human labels, check citation validity automatically, and measure abstention accuracy on unanswerable questions. In production, sample conversations for review and track user corrections and complaints.
Quiz
Pick an answer to see whether it’s right and why.
0 of 6 answered
Key terms
- Hallucination
- A confident model output that is false, made up or not supported by the given sources.
- Grounding
- Giving the model trusted source material and instructing it to answer only from it, so answers are based on facts rather than guesses.
- RAG (retrieval-augmented generation)
- Fetching relevant documents for a question and adding them to the prompt, so the model answers from your data.
- Citation
- A reference in an answer that points to the exact source passage supporting a claim.
- Faithfulness
- How well every claim in an answer is supported by the provided sources. Unsupported claims are hallucinations.
Before and after
Progress is saved in this browser only.