Skip to content
← All cheat sheets

Cheat sheet · Area 02

RAG

What RAG is: the whole pipeline

Retrieval-augmented generation (RAG) means finding the most relevant pieces of your own documents for each question and putting them into the prompt, so the model answers from your data instead of from memory.

  • RAG = retrieve relevant chunks + generate an answer from them.
  • Offline (ingestion): load → parse → chunk → embed → store. Online (query): embed question → retrieve → (rerank) → prompt → generate → cite.
  • Why: private, fresh, large knowledge + citations + fewer hallucinations, without retraining.
  • Most RAG failures are retrieval failures: wrong or missing chunks. Measure retrieval separately.
  • Long context vs RAG: small, stable docs can go straight in the prompt; big, changing or permissioned corpora need RAG.

Embeddings and cosine similarity

An embedding model turns a piece of text into a vector of numbers that captures its meaning, so texts that mean similar things end up close together, and cosine similarity measures how close two of them are; this is how RAG finds relevant chunks even when the words differ.

  • Embedding model: text → fixed-length vector (e.g. 384–3072 numbers) that captures meaning.
  • Similar meaning → vectors point the same way. 'money back' finds 'refunds' with no shared words.
  • Cosine similarity: −1 to 1 (in practice usually 0 to 1 for text); higher = more similar.
  • Use the SAME model for documents and queries; re-embed everything if you switch models.
  • Choose by: quality on YOUR data, languages, dimensions/storage, cost, speed, hosting (API vs open).

Document parsing and ingestion pipelines

Before documents can be searched, they must be turned into clean, structured text (parsing), and a repeatable ingestion pipeline must keep the search index in sync as documents are added, changed and deleted.

  • Garbage in, garbage out: most RAG quality problems start with bad parsing.
  • Parse to clean text WITH structure: headings, lists, tables (as markdown), page numbers, source URL.
  • Tools: PyMuPDF/pypdf (simple PDFs), Docling, Unstructured, LlamaParse, OCR for scans, vision models for hard layouts.
  • Ingestion pipeline: detect changes → parse → chunk → embed → upsert; delete chunks of removed documents.
  • Use content hashes to skip unchanged documents; store doc ID, version, updated_at and permissions on every chunk.

Chunking strategies

Chunking splits documents into pieces small enough to retrieve precisely but complete enough to make sense on their own, and parent-child chunking gets both by searching small pieces while handing the model the larger section around them.

  • Chunk = the unit you embed and retrieve. Typical size: 200–800 tokens, tuned on your data.
  • Too small: loses context. Too big: blurry embeddings, wasted prompt tokens, worse ranking.
  • Strategies: fixed-size + overlap, sentence/paragraph-aware, structure-aware (by heading), semantic.
  • Overlap (10–20%) keeps ideas that cross a boundary findable.
  • Parent-child: embed small child chunks for precise search, return the parent section for full context.

Vector databases, HNSW and pgvector

A vector database stores embeddings with their text and metadata and finds the nearest ones to a query in milliseconds, using approximate indexes like HNSW; pgvector adds the same ability to ordinary Postgres, and metadata filters narrow results by things like date, customer or permissions.

  • Vector DB = store vectors + text + metadata; query = 'k nearest vectors to this one, matching these filters'.
  • Exact search compares with every vector (slow at millions). ANN indexes like HNSW trade a little recall for huge speed.
  • HNSW: a layered graph of neighbours; search hops greedily toward the query. Knobs: m, ef_construction, ef_search.
  • pgvector: CREATE EXTENSION vector; a vector(1536) column; ORDER BY embedding <=> query LIMIT k; HNSW index with vector_cosine_ops.
  • Filters (tenant, date, ACL) with ANN can return too few results; use iterative scans, partitioning or pre-filtering.
  • Options: pgvector, Qdrant, Weaviate, Milvus, Pinecone, Chroma, LanceDB, Elasticsearch/OpenSearch.

BM25 keyword search and hybrid search

BM25 is the classic keyword-ranking formula that finds documents sharing the query's exact words (great for codes, names and rare terms), and hybrid search combines it with vector search so you catch both exact matches and matches by meaning.

  • BM25 scores a document higher when it contains the query's words, especially rare words, with diminishing returns for repeats and a length penalty.
  • Keyword search: great at exact codes, IDs, names, jargon. Weak at synonyms and paraphrases.
  • Vector search: great at meaning. Weak at exact codes and rare names.
  • Hybrid = run both, merge the rankings (usually reciprocal rank fusion, RRF), then rerank.
  • RRF score = Σ 1 / (k + rank), k ≈ 60. No need to make the two scores comparable.

Reranking

Reranking is a second, more careful pass over the candidates your search returned, using a model that reads the question and each candidate together, so the truly most relevant chunks end up at the top before you send them to the LLM.

  • Two stages: fast retrieval for recall (top 20–100), slow reranking for precision (keep top 3–10).
  • Bi-encoder (embeddings): query and doc embedded separately → fast, less precise.
  • Cross-encoder (reranker): reads query + doc together → slower, much more precise.
  • Options: hosted rerank APIs (e.g. Cohere, Voyage, Jina), open cross-encoders (BGE reranker, ms-marco models), or an LLM as reranker.
  • Rerankers cost latency per candidate; cap the candidate count and measure the gain.

Query rewriting, HyDE and query routing

Users' questions are often vague, follow-ups or phrased very differently from your documents, so you improve retrieval by rewriting the question into a good search query, by searching with a hypothetical answer (HyDE), and by routing each question to the right index or tool.

  • Query rewriting: turn 'how long does it last?' + chat history into 'How long does the Model X laptop battery last?'.
  • Multi-query: generate several phrasings, search with each, merge the results.
  • HyDE: have an LLM write a fake answer, embed THAT, and search with it (answers look like documents).
  • Routing: decide first where to look (product docs, orders DB, pricing table, web, no retrieval at all).
  • Each step costs an extra LLM call: use small fast models and measure the gain.

Contextual retrieval

Contextual retrieval fixes chunks that make no sense on their own by having an LLM write a short sentence of context for each chunk (which document, which company, which period) and adding it before embedding and keyword indexing.

  • Problem: 'The company's revenue grew by 3%' doesn't say WHICH company or WHEN, so it can't be found.
  • Fix: for each chunk, an LLM reads the whole document and writes 1–2 sentences of situating context; prepend it to the chunk.
  • Index the contextualised chunk in BOTH the embeddings and BM25.
  • Anthropic's 2024 tests: failed retrievals −35% (contextual embeddings), −49% (+ contextual BM25), −67% (+ reranking).
  • Cost is one LLM call per chunk at ingestion; prompt caching of the full document makes it cheap.

Citations

Citations link each claim in an answer to the exact source passage that supports it, so users can check the answer and your system can automatically catch claims that have no real support.

  • Number each source in the prompt ([1], [2]…) and ask for a citation after every claim.
  • Check citations in code: the ID must exist and (ideally) the quoted text must appear in that source.
  • Some APIs return structured citations (e.g. Anthropic's citations feature gives exact cited text and location).
  • Show sources in the UI: title, link, the quoted passage, page number.
  • A claim with no valid citation → regenerate, drop, or flag it.

GraphRAG

GraphRAG builds a knowledge graph of the entities in your documents (people, teams, products) and how they relate, so the system can answer questions that need connecting facts spread across many documents, or summarising a whole collection, which plain chunk search struggles with.

  • Knowledge graph: entities (nodes) + relationships (edges), e.g. Priya —manages→ Payments team.
  • Built by an LLM extracting entities and relations from every chunk, then merging duplicates.
  • Local search: start from entities in the question and walk the graph (multi-hop questions).
  • Global search: cluster the graph into communities, summarise each, answer 'big picture' questions from summaries.
  • Expensive to build and maintain; use when questions really need connections or corpus-wide themes.

Agentic RAG

Agentic RAG lets the model run retrieval itself as a tool, in a loop, deciding what to search for, reading the results, searching again or using other tools until it has enough to answer, instead of doing one fixed search before answering.

  • Classic RAG: one search, then answer. Agentic RAG: the model decides when, what and how often to search.
  • Retrieval becomes a tool (search_docs(query)); other tools can join (SQL, web, calculator).
  • Good for multi-part, comparative and multi-hop questions; overkill for simple lookups.
  • Always cap steps and tokens, log every search, and evaluate trajectories.

Multimodal RAG: images, tables and PDFs

Multimodal RAG makes charts, tables, diagrams and scanned pages searchable, usually by turning each into good text (markdown tables, captions written by a vision model) or by embedding page images directly, while keeping a link to the original so the answering model and the user can see it.

  • Text-only RAG silently ignores charts, images, diagrams and messy tables.
  • Approach 1: convert to text (tables → markdown, images → captions by a vision model), embed the text, keep a link to the original.
  • Approach 2: embed page images directly with a multimodal embedding model (visual document retrieval).
  • At answer time, send the original image/table (not just its caption) to a multimodal LLM for accurate details.
  • Evaluate with questions whose answers live only in charts and tables.

Measuring retrieval: Recall@k, MRR and nDCG

Retrieval metrics tell you, for a set of test questions with known relevant documents, how often the right documents are found (Recall@k), how high the first right one appears (MRR), and how good the whole ranking is (nDCG), so you can improve search with numbers instead of guesses.

  • You need a test set: questions + the IDs of the chunks/docs that answer them.
  • Recall@k = share of relevant items found in the top k. 'Did we find it at all?'
  • MRR = average of 1/rank of the FIRST relevant result. 'How soon do we find something right?'
  • nDCG@k = rewards relevant items more the higher they're ranked (and supports graded relevance). 'Is the whole order good?'
  • Measure retrieval separately from answer quality; most RAG bugs show up here first.

Capstone: build a small RAG app from scratch

You build a complete, testable RAG app (ingestion, hybrid retrieval, grounded answers with citations, an HTTP API, tests and an eval gate) that answers questions about a set of company policies, and that later lessons containerise and deploy.

  • Project shape: ingest.py (chunks) → retrieve.py (hybrid search) → generate.py (grounded, cited answer) → app.py (API).
  • Make it run offline by default (mock answers) so tests and CI never need keys.
  • Every chunk has a stable ID; golden questions list the IDs that answer them.
  • Eval gate: fail the build if Recall@3, MRR or citation validity drop below thresholds.
  • Swap TF-IDF for real embeddings and the mock for a real LLM without changing the API.