Skip to content

All lessons

Area 02 // RAGLesson 39 / 14015 min read

Query rewriting, HyDE and query routing

Users' questions are often vague, follow-ups or phrased very differently from your documents, so you improve retrieval by rewriting the question into a good search query, by searching with a hypothetical answer (HyDE), and by routing each question to the right index or tool.

Think of it like this

Analogy

You walk into a hardware shop and say "the thing for the wobbly bit". A good shop assistant rewrites that in their head ("a wall plug for a loose shelf bracket?"), maybe imagines what the answer looks like ("a pack of 8 mm plugs") to know which aisle to check, and routes you: plugs are in aisle 4, but if you'd asked about your order, they'd send you to the service desk.

That's query rewriting, HyDE and routing.

The problem it solves

Retrieval quality depends on the query, and real queries are bad search queries: short, vague, full of "it" and "that", in casual words that don't match formal documents, or about something your document index doesn't even contain (an order status, a live price). Sending the user's raw text straight to the vector database is one of the most common causes of poor RAG answers.

A small "think before you search" step fixes a surprising amount.

How it works: query rewriting

  1. Take the latest message plus recent chat history.
  2. Ask a small, fast LLM to produce a standalone search query: resolve "it/that", add the product or topic, remove chit-chat. ("how long does it last?" → "How long does the Model X laptop battery last?")
  3. Optionally expand: add synonyms or the formal term ("money back" → "refund policy").
  4. Multi-query variant: generate 3–5 different phrasings, retrieve for each, and merge with RRF.
  5. Decomposition variant: split a compound question ("compare plan A and plan B prices") into sub-queries.
  6. Search with the rewritten query, but answer the user's original question.

How it works: HyDE

  1. Ask an LLM to write a short hypothetical answer to the question, without retrieval (it may be wrong; that's fine).
  2. Embed the hypothetical answer instead of (or as well as) the question.
  3. Search with that vector. Answers are written like documents ("The battery lasts around 10 hours…"), so their embeddings land closer to the real document chunks than a short question does.
  4. Never show the hypothetical answer to the user; it's only a search key.
  5. Measure it: HyDE helps most when questions and documents are phrased very differently, and can hurt when the model's guess is wildly off.

How it works: query routing

  1. Before retrieving, classify the question: which source can answer it?
  2. Typical routes: product docs index, policy index, an orders database (via a tool), a pricing table (SQL), web search, or no retrieval at all (chit-chat, maths).
  3. Implement with simple rules (keywords, regex for order numbers), a small classifier, or an LLM call with structured output that picks a route.
  4. Route to several sources when needed and merge results.
  5. Log routes and evaluate: a wrong route means retrieval can't succeed.
Think before you search: rewrite, maybe imagine an answer, and choose where to look.

See it in code

Install
pip install scikit-learn
Needs
Python 3.10+. Rewrites and the hypothetical answer are written by hand here; in a real app a small LLM produces them.

The follow-up "how long does it last?" finds nothing; the rewrite finds the right chunk; HyDE finds it with even higher similarity. Then a simple router.

query_transforms.py
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
 
chunks = [
    "The Model X battery lasts about 10 hours of video playback.",
    "The Model X screen is a 14-inch OLED panel.",
    "Battery replacements cost 89 pounds and take 3 days.",
]
vec = TfidfVectorizer().fit(chunks)
def search(q):
    s = cosine_similarity(vec.transform([q]), vec.transform(chunks))[0]
    return round(float(s.max()), 2), chunks[s.argmax()]
 
history = "User: I'm looking at the Model X laptop."
follow_up = "how long does it last?"
 
# 1) Query rewriting: turn a vague follow-up into a standalone search query
#    (an LLM does this in real life; written by hand here).
rewritten = "How long does the Model X laptop battery last?"
print("raw     :", search(follow_up))
print("rewrite :", search(rewritten))
 
# 2) HyDE: let a model write a hypothetical answer, then search with THAT text.
hypothetical = "The Model X battery lasts around 10 hours on a full charge."
print("HyDE    :", search(hypothetical))
 
# 3) Routing: decide which index (or tool) should answer before searching.
def route(q):
    q = q.lower()
    return "orders_db" if "order" in q else "pricing_table" if "cost" in q or "price" in q else "product_docs"
for q in ["Where is my order 1234?", "What does a new battery cost?", "Is the screen OLED?"]:
    print(f"route   : {q!r} -> {route(q)}")
Output
raw     : (0.0, 'The Model X battery lasts about 10 hours of video playback.')
rewrite : (0.45, 'The Model X battery lasts about 10 hours of video playback.')
HyDE    : (0.74, 'The Model X battery lasts about 10 hours of video playback.')
route   : 'Where is my order 1234?' -> orders_db
route   : 'What does a new battery cost?' -> pricing_table
route   : 'Is the screen OLED?' -> product_docs

The raw follow-up scores 0.0 (no shared words at all): the result is just the first chunk by default. In production, the rewrite prompt is short: "Given the conversation, rewrite the user's last message as a standalone search query. Reply with the query only."

A real example

Example

An e-commerce assistant that stopped losing the thread

An online electronics shop's assistant handled single questions well but failed on follow-ups: after discussing a specific TV, "does it come in 55 inch?" retrieved random 55-inch products. Logs showed 35% of messages were follow-ups.

They added a rewrite step with a small model (about 150 ms), a router that sends order and delivery questions to their orders API and stock questions to the inventory database, and multi-query retrieval for long questions. Answer accuracy on follow-up questions rose from 58% to 87% in their eval set.

The easy way

Start here

Add one step: if the conversation has more than one turn, rewrite the last message into a standalone query with a small model before searching. Then add a simple router for the two or three obviously different question types (e.g. order status vs product questions).

When you grow: the production setup

Mature systems run rewriting, routing and multi-query as one fast structured-output call, cache rewrites, evaluate each transformation separately (Recall@k per route), and grow the router into an agentic RAG loop when questions need several searches.

Trade-offs

What you gain

  • Big recall gains for follow-ups, vague questions and mismatched vocabulary.
  • Routing sends questions to the source that can actually answer them.
  • Cheap with small models.

What it costs you

  • Extra LLM calls add latency and cost.
  • A bad rewrite or hypothetical answer can steer retrieval wrong.
  • More steps to evaluate and debug.

When not to use it

  • Single-turn search over a corpus where users already type precise queries (measure before adding steps).
  • Very tight latency budgets where an extra call isn't affordable.

Common mistakes

  • Sending raw follow-up messages ("and the price?") straight to retrieval.
  • Showing users the HyDE hypothetical answer by mistake.
  • Using a big, slow model for a simple rewrite.
  • Rewriting the question so much that the answer no longer matches what the user asked.
  • No routing, so questions about live data search static documents and fail.

Interview questions

Try answering out loud first, then open the model answer.

How do you handle follow-up questions in a RAG chatbot?

Model answer

Use the chat history to rewrite the latest message into a standalone search query with a small LLM, resolving pronouns and adding context, then retrieve with it while answering the original message. Optionally generate multiple rephrasings and fuse results. Evaluate on multi-turn test conversations, because single-turn evals won't catch this.

What is HyDE and when does it help?

Model answer

Hypothetical Document Embeddings: an LLM writes a plausible answer to the query, and that text is embedded and used for retrieval. Because the hypothetical answer resembles the style and vocabulary of real documents, it often lands closer to relevant chunks than a short question, especially for zero-shot domains or when queries and documents are phrased differently. It adds latency and can mislead retrieval if the guess is off, so validate it with evals.

How would you implement query routing?

Model answer

Define the available sources and tools, then classify each query, starting with rules for obvious cases (order IDs, greetings) and an LLM or small classifier with structured output for the rest, allowing multiple routes. Execute the chosen retrievers or tools, merge results, and log route decisions. Evaluate routing accuracy on labelled queries and track failures per route.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01A user asks "and how much is it?" after talking about a product. Why does raw retrieval fail?
  2. 02What is HyDE?
  3. 03What's the risk of HyDE?
  4. 04What does query routing decide?
  5. 05What does multi-query retrieval do?
  6. 06Rewriting adds an LLM call before retrieval. How do you keep latency low?

0 of 6 answered

Key terms

Query rewriting
Turning a user's message (often a vague follow-up) into a clear, standalone search query before retrieval.
HyDE
Hypothetical document embeddings. Searching with the embedding of a model-written hypothetical answer instead of the question.
Query routing
Deciding which index, database or tool should handle a question before retrieving.
Multi-query retrieval
Generating several phrasings of a question, retrieving for each, and merging the results to improve recall.
Retrieval
Finding the pieces of stored information most relevant to a question, for example by keyword or vector search.

Before and after

Progress is saved in this browser only.