Area 02 // RAGLesson 43 / 14014 min read
Agentic RAG
Agentic RAG lets the model run retrieval itself as a tool, in a loop, deciding what to search for, reading the results, searching again or using other tools until it has enough to answer, instead of doing one fixed search before answering.
Think of it like this
Analogy
Classic RAG is a librarian who, given your question, fetches one armful of books, hands them over, and walks away. Agentic RAG is a research assistant: they look something up, read it, realise they need last year's figures too, look those up, check a table, and only then write you an answer.
The problem it solves
Classic RAG does one retrieval with the user's question, then answers. That fails when:
- The question has several parts ("compare plan A and plan B for a team of 20").
- The second search depends on the first result ("what changed in the policy that replaced the one from 2023?").
- Some information lives in another system (a database, a calculator, the web).
- The first search simply returns nothing useful, and a reworded search would.
You need the model to plan, search, read and decide whether it has enough, which is exactly what an agent does.
How it works, step by step
- Turn retrieval into a tool:
search_docs(query: str, filters: dict)returning the top chunks with IDs (tool calling). - Add other tools if useful: SQL for structured data, a calculator, web search, a "read full document" tool.
- Give the model a goal and rules: "Answer the question using the tools. Search as many times as needed. Cite sources. Stop when you can answer or after 5 searches, and say what's missing."
- Loop: the model calls a tool → your code runs it → results go back → the model decides: search again (different query or filter), call another tool, or answer.
- Enforce stop conditions: maximum steps, maximum tokens, time budget (stop conditions).
- Answer with citations from all the chunks collected along the way.
- Route: send simple questions to classic RAG (fast, cheap) and complex ones to the agentic loop (query routing).
- Evaluate the final answers and the search trajectory (did it search sensibly?) (trajectory evals).
See it in code
- Needs
- Nothing: plain Python 3. The model's decisions are simulated so the loop runs offline; the full real loop is built in the agents capstone.
The shape of an agentic RAG loop: decide, search, collect notes, repeat, answer, all under a hard step limit.
KNOWLEDGE = {
"2025 revenue": "Revenue in 2025 was 40 million pounds.",
"2026 revenue": "Revenue in 2026 was 52 million pounds.",
}
def search(query: str) -> list[str]:
return [text for key, text in KNOWLEDGE.items() if key in query.lower()]
def fake_llm_decide(question: str, notes: list[str]) -> dict:
"""Stand-in for the model's decision each turn: search again, or answer."""
if not any("2025" in n for n in notes):
return {"action": "search", "query": "2025 revenue"}
if not any("2026" in n for n in notes):
return {"action": "search", "query": "2026 revenue"}
return {"action": "answer", "text": "Revenue grew from 40m to 52m pounds, a 30% increase."}
question = "How much did revenue grow from 2025 to 2026?"
notes = []
for step in range(1, 6): # hard cap on steps: always have one
decision = fake_llm_decide(question, notes)
if decision["action"] == "answer":
print(f"step {step}: answer -> {decision['text']}")
break
found = search(decision["query"])
notes += found
print(f"step {step}: search {decision['query']!r} -> {len(found)} result(s)")step 1: search '2025 revenue' -> 1 result(s)
step 2: search '2026 revenue' -> 1 result(s)
step 3: answer -> Revenue grew from 40m to 52m pounds, a 30% increase.In a real system, fake_llm_decide is an LLM call with a search_docs tool; the model's tool calls replace the hand-written if statements. One search with the original question ("how much did revenue grow…") would likely have retrieved only one of the two years.
A real example
Example
A due-diligence assistant for acquisitions
An M&A team asks questions like "List all contracts with change-of-control clauses and the notice periods they require". Classic RAG returned a few relevant clauses and missed most.
The agentic version first searched for "change of control" across the data room, then for each contract found, opened the full contract and searched for notice periods, then compiled a table with citations. It took 30–90 seconds per question instead of 3, which lawyers were happy with for this task. Simple questions still went through the fast classic path.
The easy way
Start here
Start with classic RAG. When evals show multi-part or comparative questions failing, add a search_docs tool and let a capable model call it in a loop with a 5-step limit. Route only those questions to the loop.
When you grow: the production setup
Production agentic RAG uses frameworks or a hand-written loop with tracing (tracing), parallel tool calls for independent searches, caching of repeated searches, per-question token budgets, permission-aware retrieval tools, and trajectory evals to catch wasteful or looping search behaviour.
Trade-offs
What you gain
- Handles multi-part, comparative and multi-hop questions.
- Recovers from a bad first search by trying again.
- Can combine documents with databases, calculators and other tools.
What it costs you
- Slower and more expensive than a single retrieval.
- Less predictable: different runs may search differently.
- Harder to evaluate and debug (needs tracing).
When not to use it
- Most questions are simple lookups: classic RAG is faster, cheaper and easier to test.
- Latency budgets are tight (real-time chat on mobile, voice).
Common mistakes
- No step or token limit, so a confused agent loops for minutes.
- Sending every question through the agentic loop.
- Not logging the searches, making failures impossible to diagnose.
- Search tools that ignore user permissions.
- Evaluating only final answers and never the search trajectory.
Interview questions
Try answering out loud first, then open the model answer.
What is agentic RAG and how does it differ from standard RAG?
Model answer
Standard RAG runs a fixed pipeline: one retrieval with the (possibly rewritten) query, then generation. Agentic RAG exposes retrieval (and other tools) to an LLM agent that iteratively decides what to search, inspects results, refines queries, uses other tools and decides when it has enough evidence, under step and budget limits. It handles complex, multi-step questions at the cost of latency, cost and variability.
How would you control cost and reliability in agentic RAG?
Model answer
Route only complex queries to the agent, cap steps, tokens and wall-clock time, use a small model for simple decisions, run independent searches in parallel, cache tool results, return concise results from tools, trace every step, and evaluate both final answers and trajectories. Fall back to a best-effort answer stating what's missing when limits are hit.
When would you choose agentic RAG over GraphRAG for multi-hop questions?
Model answer
When relationships are varied or unpredictable, the corpus changes often, or multi-hop questions are a minority, so building and maintaining a graph isn't worth it. GraphRAG fits when the same kinds of relationships are queried constantly and need fast, consistent answers. Agentic RAG trades higher per-query cost for zero ingestion-time graph building.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Agentic RAG
- RAG where the model uses retrieval as a tool in a loop, deciding what and how often to search before answering.
- Agent
- A system where a model decides, in a loop, which tools to call and what to do next until a task is done.
- Tool calling (function calling)
- A model replying with a request to run one of your functions, with arguments; your app runs it and sends the result back.
- Retrieval
- Finding the pieces of stored information most relevant to a question, for example by keyword or vector search.
- Stop condition
- A rule that ends an agent loop, such as a final answer, a maximum number of steps, a token or time budget, or an error.
Before and after
Progress is saved in this browser only.