Skip to content

All lessons

Area 02 // RAGLesson 46 / 14030 min read

Capstone: build a small RAG app from scratch

You build a complete, testable RAG app (ingestion, hybrid retrieval, grounded answers with citations, an HTTP API, tests and an eval gate) that answers questions about a set of company policies, and that later lessons containerise and deploy.

Think of it like this

Analogy

Building a first RAG app is like setting up a small library with a helpful librarian:

  1. Shelve the books (ingest): every policy goes on the shelf, split into labelled sections.
  2. Learn the catalogue (retrieve): a librarian who can find sections by exact words and by meaning.
  3. Answer at the desk (generate): read the right sections and answer, pointing at the page.
  4. Open the doors (API): people can ask from anywhere.
  5. Inspect the library (tests and evals): every week, check the librarian still finds the right sections.

The problem it solves

You've learned each RAG piece separately. Real value comes from wiring them into something that runs, can be tested, can be measured and can be deployed. Many tutorials stop at a notebook that calls an API with a key and prints an answer, which you can't test in CI, can't measure, and can't safely change.

This capstone builds a small app with the shape of a production system, so the Deployment area can containerise it and put it behind a CI/CD pipeline with an eval gate.

How it works, step by step

The full project lives in the course repository at examples/rag-app/. It answers employees' questions about four policy documents (leave, expenses, IT security, remote work).

  1. Project layout:
examples/rag-app
rag_app/
  ingest.py     # markdown -> chunks with stable IDs and "Doc > Section" titles
  retrieve.py   # hybrid search: BM25 + vector-style search, fused with RRF
  generate.py   # grounded, cited answer (mock | anthropic | openai | ollama)
  app.py        # FastAPI: GET /health, POST /ask
data/docs/      # the policy documents (markdown)
tests/          # fast tests, no API keys
evals/          # golden.jsonl + run_eval.py (the eval gate)
requirements.txt, requirements-dev.txt
  1. Ingest (parsing, chunking): read every markdown file, make one chunk per ## section, and give it a stable ID (leave#2) plus a title (Leave policy > Carrying over leave) that's indexed with the text.
  2. Retrieve (hybrid search): BM25 over words, plus a vector-style index (TF-IDF here so it runs anywhere; swap in real embeddings later), merged with reciprocal rank fusion. Return the top 3 chunks.
  3. Generate (grounding, citations): number the chunks as sources, tell the model to answer only from them, cite [n] after every sentence, and say "I don't know" otherwise. LLM_PROVIDER chooses Claude, OpenAI, a local Ollama model, or an offline mock that quotes the best source.
  4. Check citations: every [n] must point at a real source; the API returns citations_valid.
  5. Serve (FastAPI, Pydantic): POST /ask validates the question (3–500 characters) and returns the answer, sources and the citation check; GET /health reports status, chunk count and version.
  6. Test: six fast tests cover chunking, retrieval, the API and citation checking, with no keys.
  7. Evaluate (retrieval metrics, golden datasets): evals/golden.jsonl holds 12 real questions with the chunk IDs that answer them. run_eval.py computes Recall@3, MRR and citation validity, and exits with code 1 if any falls below its threshold, ready to become a CI eval gate.
Architecture
Loading diagram…
The capstone app. Ingestion runs at start-up here; a bigger app would run it as a separate job.
  • data/docs/*.md to ingest.py
  • ingest.py to retrieve.py
  • Client to app.py (FastAPI)
  • app.py (FastAPI) to retrieve.py
  • retrieve.py to generate.py
  • generate.py to app.py (FastAPI)
  • retrieve.py to evals/run_eval.py
  • generate.py to evals/run_eval.py

See it in code

Install
pip install -r examples/rag-app/requirements-dev.txt
Needs
Python 3.11+. No API key: the default mock answers offline. Set LLM_PROVIDER and a key to use a real model.

The heart of the app: the API endpoint that ties retrieval, generation and the citation check together (from rag_app/app.py).

rag_app/app.py (excerpt)
from fastapi import FastAPI
from pydantic import BaseModel, Field
 
from .generate import cited_ids, generate
from .ingest import load_corpus
from .retrieve import Retriever
 
CHUNKS = load_corpus(DOCS)               # ingest once at start-up (fine for a small corpus)
RETRIEVER = Retriever(CHUNKS)
app = FastAPI(title="Policy RAG", version=VERSION)
 
class Question(BaseModel):
    question: str = Field(min_length=3, max_length=500)   # validated before we spend tokens
 
@app.get("/health")
def health() -> dict:
    return {"status": "ok", "chunks": len(CHUNKS), "version": VERSION}
 
@app.post("/ask", response_model=Answer)
def ask(q: Question) -> Answer:
    chunks = RETRIEVER.search(q.question, k=3)             # hybrid retrieval
    text = generate(q.question, chunks)                    # grounded + cited answer
    _, valid = cited_ids(text, len(chunks))                # every [n] must be a real source
    sources = [Source(n=i, id=c.id, title=c.title) for i, c in enumerate(chunks, start=1)]
    return Answer(answer=text, sources=sources, citations_valid=valid)

Run the tests, the eval gate and the server:

Terminal
cd examples/rag-app
pytest -q
python -m evals.run_eval
uvicorn rag_app.app:app --reload        # API docs at http://127.0.0.1:8000/docs
Example output (eval gate, then POST /ask with the offline mock)
......                                                                   [100%]
6 passed in 2.32s
PASS  recall@3         1.00  (threshold 0.90)
PASS  mrr              0.96  (threshold 0.75)
PASS  citations_valid  1.00  (threshold 1.00)
12 questions, 12 chunks
 
{
  "answer": "Report a lost or stolen laptop or phone to the IT helpdesk within 1 hour, by calling extension 4357. [1]",
  "sources": [
    {"n": 1, "id": "it-security#2", "title": "IT security rules > Lost or stolen devices"},
    {"n": 2, "id": "it-security#3", "title": "IT security rules > Using AI tools"},
    {"n": 3, "id": "expenses#1", "title": "Expenses policy > What you can claim"}
  ],
  "citations_valid": true
}

With LLM_PROVIDER=anthropic (or openai, ollama), the same endpoint returns a written answer from a real model, still grounded in the three sources and cited.

A real example

Example

From capstone to a real internal tool

A five-person startup used almost exactly this shape for its first internal assistant. They swapped TF-IDF for an embedding model, the markdown loader for a Notion export, and the mock for a small, cheap model, without touching the API or tests. The golden set grew from 12 to 120 questions taken from what people actually asked in Slack.

When someone later "improved" the chunking, the eval gate failed with Recall@3 dropping from 0.91 to 0.78, and the change never reached production. That one red CI run paid for all the setup.

The easy way

Start here

  1. Copy examples/rag-app, replace data/docs with your own markdown files.
  2. Write 10 questions in evals/golden.jsonl with the chunk IDs that answer them.
  3. Run pytest and python -m evals.run_eval until both pass.
  4. Set LLM_PROVIDER=ollama (free) or your provider, and try real answers.

When you grow: the production setup

The production version adds a real embedding model and a vector database (pgvector), an ingestion job with change detection, a reranker, query rewriting, permission filters, streaming answers (SSE), tracing (Langfuse), LLM-judged faithfulness evals, and the Docker image, CI/CD pipeline, health checks and rollback from the Deployment capstone.

Trade-offs

What you gain

  • Every stage is small, testable and swappable behind its own module.
  • Runs offline by default, so tests and CI are fast, free and reliable.
  • Measured from day one: retrieval and citation quality are numbers, not feelings.

What it costs you

  • TF-IDF misses paraphrases that real embeddings would catch.
  • Ingesting at start-up doesn't scale to large or changing corpora.
  • The mock generator only quotes; real answer quality needs a real model and LLM-judged evals.

When not to use it

  • Don't copy this shape for corpora of millions of documents without moving ingestion to a separate job and the index to a real vector database.

Common mistakes

  • Building the first version in a notebook with a hard-coded API key, so it can't be tested or deployed.
  • No stable chunk IDs, so citations and evals can't refer to the same thing.
  • Only testing "does it return 200?" instead of measuring retrieval quality.
  • Skipping the "I don't know" path, so unanswerable questions get invented answers.
  • Letting the golden set go stale as documents are added.

Interview questions

Try answering out loud first, then open the model answer.

Walk me through how you'd structure a production-ready RAG service.

Model answer

Separate modules for ingestion (parse, chunk with stable IDs and metadata, embed, upsert), retrieval (hybrid search, filters, reranking behind one interface), generation (prompt template with numbered sources, grounding and abstention rules, provider adapter, citation validation) and the API layer (validated requests, health endpoint, streaming). Configuration via environment variables, an offline mode for tests, unit and API tests, a golden eval set with a CI gate, tracing and metrics, and a container image deployed through CI/CD.

How do you test a RAG application?

Model answer

Unit tests for deterministic parts (chunking, ID stability, prompt building, citation parsing), API tests with a test client, retrieval evals on a golden set (Recall@k, MRR) that run in CI with thresholds, and end-to-end answer evals (faithfulness, relevance, abstention on unanswerable questions) with an LLM judge on a schedule or before release. Keep the model behind a mockable adapter so most tests need no keys.

How would you make this app handle a much larger and changing document set?

Model answer

Move ingestion out of start-up into a scheduled or event-driven job with change detection and deletes, store vectors in a vector database or pgvector with metadata and permission filters, switch to a real embedding model and add a reranker, add caching, and scale the stateless API horizontally behind a load balancer. Keep the eval set growing and run it on every change.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01Why does the app default to a mock answer generator?
  2. 02Why give every chunk a stable ID like leave#2?
  3. 03What does the eval gate do?
  4. 04Which endpoint lets Docker or a load balancer check the app is alive?
  5. 05You want real embeddings instead of TF-IDF. What must change?
  6. 06A new policy document is added. What else should you add?

0 of 6 answered

Key terms

RAG (retrieval-augmented generation)
Fetching relevant documents for a question and adding them to the prompt, so the model answers from your data.
Chunk
A small piece of a document (often a few hundred tokens) that is embedded, stored and retrieved as one unit.
Hybrid search
Running keyword search and vector search together and merging their results, to catch both exact terms and meaning.
Citation
A reference in an answer that points to the exact source passage supporting a claim.
Eval
A repeatable test of an AI system's quality on a set of examples with a scoring method.
FastAPI
A popular Python web framework for building APIs, using type hints and Pydantic for validation and automatic docs.

Before and after

Read first

Read next

Progress is saved in this browser only.