Area 02 // RAGLesson 40 / 14013 min read
Contextual retrieval
Contextual retrieval fixes chunks that make no sense on their own by having an LLM write a short sentence of context for each chunk (which document, which company, which period) and adding it before embedding and keyword indexing.
Think of it like this
Analogy
Photocopy one page from the middle of a company report and hand it to someone: "Revenue grew by 3% over the previous quarter." Which company? Which quarter? They can't tell. Now write a sticky note on top: "From ACME Corp's report for Q2 2023." Suddenly the page is useful on its own.
Contextual retrieval writes that sticky note on every chunk before it goes into the index.
The problem it solves
Chunking cuts documents into pieces, and pieces lose context: the company name was in the title, the year in the header, the product name three paragraphs earlier. A question like "How did ACME's revenue change in Q2 2023?" can't match a chunk that never mentions ACME or Q2. Worse, it may match a similar chunk from a different company's report and produce a confidently wrong answer.
Adding the heading path helps, but many chunks need more specific context than headings give.
How it works, step by step
- Chunk the document as usual.
- For each chunk, call an LLM with the whole document and the chunk, asking for a short context: "Here is the document… Here is a chunk from it… Write 1–2 sentences that situate this chunk within the document, to improve search retrieval. Answer only with the context."
- Prepend the context to the chunk: "This chunk is from ACME Corp's SEC filing for Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3%…"
- Index the contextualised chunk in both the embeddings and the BM25 index.
- Retrieve as usual (hybrid search, then reranking).
- Keep costs down with prompt caching: the whole document is the same for every chunk of that document, so cache it and only the small chunk part changes per call. Use a small, cheap model.
- Measure the gain: Anthropic reported (September 2024) that across their test sets, top-20 retrieval failures fell by 35% with contextual embeddings, 49% when BM25 was contextualised too, and 67% with reranking added.
See it in code
- Install
pip install scikit-learn- Needs
- Python 3.10+. The context sentences are written by hand here; at ingestion a small LLM writes them.
Two companies' filings contain nearly identical sentences. Without context, the search picks the wrong company; with context, it picks the right one.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
chunks = [
"The company's revenue fell by 2% over the previous quarter.", # from Globex's Q2 filing
"The company's revenue grew by 3% over the previous quarter.", # from ACME's Q2 filing
"Headcount stayed flat at 1,200 employees.", # from ACME's Q2 filing
]
# Contextual retrieval: an LLM writes 1 sentence of context per chunk, from the whole document.
context = [
"This chunk is from Globex Inc's SEC filing for Q2 2023.",
"This chunk is from ACME Corp's SEC filing for Q2 2023.",
"This chunk is from ACME Corp's SEC filing for Q2 2023.",
]
contextual = [f"{c} {chunk}" for c, chunk in zip(context, chunks)]
query = "How did ACME revenue change in Q2 2023?"
for name, texts in [("plain chunks ", chunks), ("contextual chunks", contextual)]:
vec = TfidfVectorizer().fit(texts + [query])
scores = cosine_similarity(vec.transform([query]), vec.transform(texts))[0]
best = scores.argmax()
print(f"{name}: best = chunk {best} {context[best].split()[4]} ({scores[best]:.2f})")plain chunks : best = chunk 0 Globex (0.06)
contextual chunks: best = chunk 1 ACME (0.17)Without context, the two revenue sentences look the same to search and the wrong company wins. With one sentence of context, ACME's chunk wins clearly. The context-writing call in production is a normal LLM request with the document in a cached system prompt and the chunk in the user message.
A real example
Example
Financial research across 5,000 earnings reports
An investment research tool indexed thousands of quarterly reports. Analysts' questions always named a company and period, but most chunks didn't: "Gross margin improved due to lower freight costs." Retrieval often returned the right kind of sentence from the wrong company.
They added contextual retrieval at ingestion with a small model and prompt caching (each report cached once, then every chunk contextualised). Ingestion cost rose modestly; the share of answers citing the wrong company dropped sharply in their eval set, and analysts stopped double-checking every citation.
The easy way
Start here
Before going full contextual retrieval, try the cheap version: prefix every chunk with its document title and heading path. If chunks still lose key context (company, product, period, version), add the LLM-written context step with a small model and prompt caching.
When you grow: the production setup
At scale, contextualisation runs inside the ingestion pipeline as a batch job (often through a batch API at half price), stores the context separately from the original chunk (so it can be regenerated or improved), versions the context prompt, and is evaluated with retrieval metrics per document type.
Trade-offs
What you gain
- Large retrieval gains for chunks that lose context when cut out.
- Works with any embedding model and with BM25.
- One-time cost at ingestion; no extra latency at query time.
What it costs you
- An LLM call per chunk: ingestion becomes slower and costs more.
- Re-running is needed when documents change.
- Generated context can occasionally be wrong; it should be checked on samples.
When not to use it
- Chunks are already self-contained (FAQ entries, product cards, short tickets).
- Titles and heading paths already supply the missing context (measure first).
Common mistakes
- Only contextualising the embeddings and not the BM25 index.
- Sending the whole document uncached for every chunk, multiplying costs.
- Using a large, slow model for a simple summarisation step.
- Letting the context sentence grow into a long summary that drowns the chunk.
- Not evaluating whether it actually improved retrieval on your data.
Interview questions
Try answering out loud first, then open the model answer.
What is contextual retrieval?
Model answer
An ingestion-time technique: for every chunk, an LLM reads the full source document and writes a brief context situating the chunk (entity, period, section), which is prepended before embedding and BM25 indexing. It recovers information lost by chunking, improving recall and reducing wrong-entity matches. Prompt caching the document makes it affordable, and combining it with hybrid search and reranking compounds the gains.
How would you keep the cost of contextual retrieval under control?
Model answer
Use a small, cheap model; cache the document prefix so each per-chunk call only pays full price for the chunk; run it as a batch job (batch APIs halve cost); skip chunks that are already self-contained; only reprocess changed documents; and cap the context length. Measure the retrieval gain per pound spent.
How does contextual retrieval compare with parent-child chunking?
Model answer
Parent-child retrieves by small chunks but sends larger parent sections to the LLM; it helps the answer step but doesn't help a context-less chunk get found. Contextual retrieval changes what gets indexed, so the chunk itself becomes findable. They solve different parts of the problem and combine well.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Contextual retrieval
- Adding an LLM-written sentence of context to each chunk (which document, entity, period) before embedding and keyword indexing.
- Chunk
- A small piece of a document (often a few hundred tokens) that is embedded, stored and retrieved as one unit.
- Prompt caching
- A provider feature that reuses the processed beginning of a prompt when it repeats exactly, making those tokens much cheaper and faster.
- BM25
- The classic keyword-ranking formula. Scores documents by the query words they contain, favouring rare words, with saturation and length normalisation.
- Embedding
- A vector of numbers that represents the meaning of a piece of text (or an image), so that similar meanings get similar vectors.
Before and after
Progress is saved in this browser only.