What RAG is: the whole pipeline

How it works: the RAG pipeline end to end

RAG has two halves: an offline ingestion pipeline that prepares your documents, and an online query pipeline that runs for each question.

Offline: ingestion (runs when documents are added or change)

1. Load documents from where they live: PDFs, web pages, Google Drive, Notion, a database, support tickets.
2. Parse them into clean text with structure (headings, tables, page numbers) (document parsing).
3. Chunk the text into pieces of a few hundred tokens, each a self-contained idea (chunking).
4. Embed each chunk with an embedding model, turning it into a vector of meaning (embeddings).
5. Store the vectors with their text and metadata (source, date, owner, permissions) in a vector database or a search engine.

Online: query (runs for every question)

6. Embed the question with the same embedding model.
7. Retrieve the most similar chunks (often with keyword search too: hybrid search), filtered by metadata and the user's permissions.
8. Rerank the candidates with a more accurate model and keep the best few (reranking).
9. Build the prompt: instructions ("answer only from the sources, cite them, say if unknown") + the chunks + the question.
10. Generate the answer with the LLM, with grounding in the sources.
11. Cite and check: show which sources were used and verify the citations (citations).

How it works: long context vs RAG

As of November 2026 many models accept around 1M tokens, so "just paste everything in" is sometimes a real option. Choose with these questions:

1. How big is the knowledge? A few documents or one long contract (under a few hundred thousand tokens): long context can work. Thousands of documents: RAG.
2. How often does it change? Long context needs you to resend the latest version anyway; RAG updates one chunk when one paragraph changes.
3. How many questions per document? Many questions about the same document → long context plus prompt caching is attractive.
4. Who may see what? Per-user permissions are much easier to enforce at retrieval time (access control in RAG).
5. Cost and latency? Sending 500K tokens on every question is slow and expensive; RAG sends a few thousand.
6. Quality? Very long contexts can bury the relevant part (lost in the middle); good retrieval puts the right text front and centre.

Many systems combine both: retrieve generously (say 20–50 chunks or whole sections) into a large context window.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
 
# 1) INGEST: your documents (in real life: PDFs, web pages, tickets…)
docs = {
    "refunds.md": "Refunds are available within 30 days. Opened software cannot be refunded.",
    "shipping.md": "Standard shipping takes 3-5 working days. Express shipping takes 1 day.",
    "warranty.md": "All laptops have a 2-year warranty covering hardware faults.",
}
# 2) CHUNK: split into small pieces (here: one sentence each)
chunks = [(name, s.strip() + ".") for name, text in docs.items() for s in text.split(".") if s.strip()]
 
# 3) EMBED + 4) STORE: turn every chunk into a vector (TF-IDF here; an embedding model in real life)
vectorizer = TfidfVectorizer().fit([c for _, c in chunks])
index = vectorizer.transform([c for _, c in chunks])
 
# 5) RETRIEVE: embed the question the same way and take the most similar chunks
question = "How long does express shipping take?"
scores = cosine_similarity(vectorizer.transform([question]), index)[0]
top = scores.argsort()[::-1][:2]
 
# 6) GENERATE: put the retrieved chunks into the prompt for the LLM
context = "\n".join(f"[{i + 1}] ({chunks[j][0]}) {chunks[j][1]}" for i, j in enumerate(top))
prompt = f"Answer using only the sources below and cite them.\n\n{context}\n\nQuestion: {question}"
print(prompt)

Answer using only the sources below and cite them.
 
[1] (shipping.md) Express shipping takes 1 day.
[2] (shipping.md) Standard shipping takes 3-5 working days.
 
Question: How long does express shipping take?