Query-Side Optimization: Rewriting, Hypothetical Document Embeddings, Multi-Query, Step-Back Prompting, and Intent Routing

Query rewriting: completing what was left unsaid

Query rewriting is the plainest and most valuable of the four. It does three things: resolving references, splitting compound questions, and turning colloquialism into retrieval-friendly phrasing.

Resolving references: the last turn asked who must approve a production database failover in writing, and this turn asks what that person's name is. "That person" sent to BM25 fetches every chunk discussing people. The rewriter must turn it into "what is the platform team lead's name" — with the antecedent coming from the previous turn.

Splitting compound questions: "how many annual leave days carry over and by when must the carried-over portion be used" is two questions, and combined, the two topics' terms dilute each other and neither ranks.

Colloquial to retrieval-friendly: in "could you tell me roughly what that reimbursement limit is," the filler words appear everywhere in the corpus with near-zero inverse document frequency, serving only to lengthen the query and skew BM25's length normalization.

In implementation it is one very short model call. Two details: set temperature to 0, or the same sentence rewrites differently each time and evaluation numbers drift; and explicitly forbid the model from answering the question, or it helpfully writes the answer in too, and that passage sent to retrieval is pure pollution.

const REWRITE_SYSTEM = [
  'You are a retrieval system query rewriter. Rewrite the user\'s current question into a search query.',
  '1. Resolve references from the conversation history (this person, he, that one become specific names); leave as-is when it cannot be resolved.',
  '2. Remove filler words and pleasantries, keeping only content words with retrieval value.',
  '3. Do not answer the question and do not add facts absent from the history.',
  'Output only the rewritten line, with no explanation.',
].join('\n')
 
export async function rewriteQuery(client, history, question) {
  const context = history
    .slice(-4)
    .map((t) => `${t.role === 'user' ? 'User' : 'Assistant'}: ${t.content}`)
    .join('\n')
  const res = await client.messages.create({
    model: 'claude-haiku-4-5-20251001', // rewriting is a short task; the cheap fast tier suffices
    max_tokens: 512,
    temperature: 0, // one sentence must rewrite identically, or evaluation numbers drift
    system: REWRITE_SYSTEM,
    messages: [{ role: 'user', content: `History:\n${context || '(none)'}\n\nThis turn: ${question}` }],
  })
  return res.content.find((b) => b.type === 'text').text.trim()
}

REWRITE_SYSTEM = "\n".join([
    "You are a retrieval system query rewriter. Rewrite the user's current question into a search query.",
    "1. Resolve references from the conversation history; leave as-is when it cannot be resolved.",
    "2. Remove filler words and pleasantries, keeping only content words with retrieval value.",
    "3. Do not answer the question and do not add facts absent from the history.",
    "Output only the rewritten line, with no explanation.",
])
 
 
def rewrite_query(client, history, question: str) -> str:
    context = "\n".join(
        f"{'User' if turn['role'] == 'user' else 'Assistant'}: {turn['content']}"
        for turn in history[-4:]
    )
    res = client.messages.create(
        model="claude-haiku-4-5-20251001",  # rewriting is a short task; the cheap fast tier suffices
        max_tokens=512,
        temperature=0,  # one sentence must rewrite identically, or evaluation numbers drift
        system=REWRITE_SYSTEM,
        messages=[{
            "role": "user",
            "content": f"History:\n{context or '(none)'}\n\nThis turn: {question}",
        }],
    )
    return next(b.text for b in res.content if b.type == "text").strip()

Three bills: one extra model call per question, with one line of output on the cheap fast tier; one extra round trip of latency; and the metric barely moves on single-turn question answering — the lab's metrics are identical with it on or off. Its value is entirely in multi-turn, expanded at the end of this section.

Hypothetical document embeddings: invent an answer first and retrieve on that

The most counter-intuitive of the four, known as HyDE: have the model invent an answer to the question first, then retrieve using that invented answer rather than the question. Hearing it first sounds absurd — what the model invents may be entirely wrong, so how does retrieving on something wrong help?

The key is that question and answer do not live in the same semantic space. A user asks what the nightly accommodation cap is for a first-tier city, which is an interrogative; the corpus's passage reads "the nightly accommodation standard for first-tier cities is such-and-such, with any excess requiring the department head's signature" — declarative, with concrete numbers, in official register. Two passages about the same thing, entirely different in register, syntax, and vocabulary, and vector search compares exactly semantic similarity.

HyDE turns the interrogative into a stretch of fake text that looks like a policy clause: "Regarding first-tier city accommodation standards, per company policy, the nightly cap is such-and-such." The numbers in it may be invented, and that is fine — we never show it to the user and only use its vector to find neighbors. It sits in the same register with the same official vocabulary as the real passage, and is naturally much closer.

Three bills: one extra model call per question, and the most expensive of the four — a hypothetical document runs 80 to 150 words, more than ten times the rewriter's output tokens; retrieval count doubles; and our offline lab cannot answer the metric bill. In offline mode this step is a rule-based stand-in, only layering register words on, unable to invent something like a concrete cap — which is exactly where HyDE works. Offline movements show only that the path connects and prove neither that HyDE helps nor that it does not. This bill must be run yourself against a real model.

Multi-query: one question, several angles, results fused by reciprocal rank

Multi-query's idea is simple: the documents may use a different word for the same thing. A user asks what process a consumer scale-up follows, and the meeting note says it is a class-two change requiring a ticket at the change review board per the standard. Scale-up and change, process and standard — not one character overlaps.

So have the model generate two or three differently angled search queries: one keyword-leaning (nouns and numbers only), one document-register-leaning, plus the original question, each searched and then fused into one list.

Fusion uses day nine's reciprocal rank fusion with a constant of 60 and an id tie-break for reproducibility — the principle is on day nine and the same function is reused. Note one thing: day nine fused the ranks of two different retrievers, and today fuses the ranks of one retriever over different queries. The axis of fusion changed and the function did not — precisely what makes it better than weighting by score: it does not look at scores at all.

const RRF_K = 60 // the value common in the original paper, uniform across this course
 
export function rrfFuse(rankings, k = RRF_K) {
  const scores = new Map()
  for (const ranking of rankings) {
    ranking.forEach((id, index) => {
      // Ranks start at 1
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + index + 1))
    })
  }
  return [...scores.entries()]
    .map(([id, score]) => ({ id, score }))
    .sort((a, b) => b.score - a.score || (a.id < b.id ? -1 : 1)) // tie-break by id for reproducibility
}
 
export async function multiQuerySearch(retriever, variants) {
  const rankings = []
  for (const q of variants) {
    const hits = await retriever.retrieve(q) // one search per variant, the fastest-growing cost here
    rankings.push(hits.map((h) => h.chunkId))
  }
  return rrfFuse(rankings)
}

RRF_K = 60  # the value common in the original paper, uniform across this course
 
 
def rrf_fuse(rankings: list[list[str]], k: int = RRF_K) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for index, doc_id in enumerate(ranking):  # ranks start at 1
            scores[doc_id] = scores.get(doc_id, 0.0) + 1 / (k + index + 1)
    # Tie-break by id for reproducibility: Python's sort is stable and both keys sort in one pass
    return sorted(scores.items(), key=lambda kv: (-kv[1], kv[0]))
 
 
async def multi_query_search(retriever, variants: list[str]) -> list[tuple[str, float]]:
    rankings = []
    for q in variants:
        hits = await retriever.retrieve(q)  # one search per variant, the fastest-growing cost here
        rankings.append([h.chunk_id for h in hits])
    return rrf_fuse(rankings)

Three bills: one extra model call per question (generate all variants in one call, not one call per variant); retrieval count multiplied by the variant count, so three variants is triple; and the metric is a net loss — recall did not move and nDCG fell from 0.671 to 0.665. A thirty-document corpus has no long tail requiring a different angle to fetch, and the two extra paths merely reshuffled the same results. Retest at ten thousand documents and the conclusion may well invert.

Step-back prompting: ask a more general question first, get the background, come back

Step-back prompting targets a different failure: a question asked so specifically that the corpus contains no such sentence.

A user asks whether temporarily raising a workspace's trace sampling rate to one hundred percent is a class-something change and how many working days in advance a ticket must be filed. No passage in the corpus discusses what class a sampling rate change is — change classification is in one policy document and sampling rates in a monitoring document, and retrieving this long question directly has both sides' terms diluting each other so that neither ranks.

Step back and ask: what are the change review classification rules and ticket lead times? That is a more general question, and the corpus happens to have a whole section on it; get the background and return to the specific question, and the answer is in the background.

Implementation is isomorphic to multi-query and differs only in the prompt — multi-query wants different phrasings at the same level and step-back wants the level above's overview.

Three bills: one extra model call per question (one line of output); retrieval count doubles; and the metric bill shares HyDE's affliction, since it too is a rule-based stand-in offline. But it has one advantage HyDE lacks: its output is short, so whatever its effect, it is the cheapest of the four expansion techniques.

Intent routing: not every question needs retrieval

The four techniques above all optimize retrieving more accurately. Intent routing asks a different question: does this one need retrieval at all?

A user says "hello" or "thanks, what else can you look up for me," and no amount of retrieval finds an answer. But if the pipeline sends every input through retrieval, those still trigger vectorization, a database query, and context assembly, and then stuff a pile of irrelevant chunks into the prompt from which the model has to struggle free to reply "hello." Money spent and accuracy down.

So add a very light classifier at the front of the pipeline, splitting three ways:

• Answer directly: chat, greetings, questions about the system's capabilities. Zero retrievals.
• Single-hop retrieval: one document holds the answer. The standard flow.
• Multi-hop retrieval: an intermediate fact must be found first (who holds a role, which standard applies), then used to search a second document. Two rounds of retrieval.

The classifier can be very cheap — one short call outputting one of three words.

const INTENT_SYSTEM = [
  'You are a retrieval system intent classifier. Decide which path this question takes, outputting one word:',
  'direct     — greetings, thanks, or asking what you can do, where searching the knowledge base is pointless;',
  'single-hop — one document holds the answer;',
  'multi-hop  — an intermediate fact must be found first and used to search a second document.',
].join('\n')
 
export async function route(client, retriever, query, budgets) {
  const res = await client.messages.create({
    model: 'claude-haiku-4-5-20251001',
    max_tokens: 16, // one word only; do not give it room to write an essay
    temperature: 0,
    system: INTENT_SYSTEM,
    messages: [{ role: 'user', content: query }],
  })
  const text = res.content.find((b) => b.type === 'text').text
  // Undecidable falls back to single-hop: the fallback must lean toward spending a little more, never toward not answering
  const intent = text.includes('direct') ? 'direct' : text.includes('multi') ? 'multi-hop' : 'single-hop'
  if (intent === 'direct') return { intent, candidates: [], budget: 0 }
  const candidates = await retriever.retrieve(query)
  // Multi-hop must fit two documents, and 600 tokens often fits only the first — routing is not only about saving money
  return { intent, candidates, budget: intent === 'multi-hop' ? budgets.multiHop : budgets.default }
}

INTENT_SYSTEM = "\n".join([
    "You are a retrieval system intent classifier. Decide which path this question takes, outputting one word:",
    "direct     — greetings, thanks, or asking what you can do;",
    "single-hop — one document holds the answer;",
    "multi-hop  — an intermediate fact must be found first and used to search a second document.",
])
 
 
async def route(client, retriever, query: str, budgets: dict) -> dict:
    res = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=16,  # one word only; do not give it room to write an essay
        temperature=0,
        system=INTENT_SYSTEM,
        messages=[{"role": "user", "content": query}],
    )
    text = next(b.text for b in res.content if b.type == "text")
    # Undecidable falls back to single-hop: lean toward spending more, never toward not answering
    intent = "direct" if "direct" in text else "multi-hop" if "multi" in text else "single-hop"
    if intent == "direct":
        return {"intent": intent, "candidates": [], "budget": 0}
    candidates = await retriever.retrieve(query)
    # Multi-hop must fit two documents, and 600 tokens often fits only the first
    budget = budgets["multi_hop"] if intent == "multi-hop" else budgets["default"]
    return {"intent": intent, "candidates": candidates, "budget": budget}

Note that comment about multi-hop needing to fit two documents. Day eight's context budget is 600 tokens, ample for a single-document question and often enough for only the first document of a multi-hop one. Recognizing multi-hop and giving it a higher budget measurably took multi-hop recall from 50% to 100%. So routing pays off twice: you stop spending where you should not, and you still have budget left where you should.

Those two originally failing multi-hop questions are worth taking apart, because they broke in different places and were fixed by different mechanisms. The first's second answer document never entered the candidate pool — the two documents share not one character, and no amount of reordering conjures what was not fetched; the bridging round retrieving on "platform team lead" is what brought it in. The second is the opposite: that document already ranked 4th in the baseline, and neither path's raw score cleared the gate (0 on keyword, 0.151 on vector, just short), so it was blocked by the refusal gate; the bridging round searching on the change review standard gave the same document a high keyword score, so it cleared the gate on its own raw score and rose to 1st along the way. Not in the candidate pool and in the pool but blocked are two ailments, and one medicine does not treat both.

How the fallback is designed: undecidable falls back to single-hop. Judging multi-hop as single-hop only searches one round fewer and answers incompletely, and judging chat as single-hop only wastes one retrieval; conversely, judging something needing retrieval as direct leaves the model with no material and only invention, the most expensive error. The fallback must lean toward spending a little more and never toward not answering.

Multi-turn conversation: without resolving references, retrieval will go astray

Query rewriting barely moves the metric on single-turn question answering, and the moment multi-turn arrives it goes from a nicety to the line between working and not. The lab runs this conversation:

Turn one asks who must approve a production database failover in writing, hitting the on-call procedure document stating it must be the platform team lead in writing. Turn two asks what that person's name is.

Retrieving those five words unresolved gives, measurably: not one candidate clears the gate and the system can only refuse. Note the failure mode — not answering wrongly but saying it cannot find an answer that is plainly in the corpus. A user who just asked and is told on the next line that nothing was found experiences a collapse of trust.

After rewriting, the sentence becomes "what is the platform team lead's name." Same retriever, same index, and this time the organization chart document enters the context — only that document states who the platform team lead is. The question went from unanswerable to answerable, and we changed only that sentence.

That also raises a deeper point: the refusal itself was correct (retrieval genuinely fetched nothing), and the reason was not that the corpus lacks the answer but that we asked the question badly. Generation-side refusal policy was settled on day six and governs having material insufficient to support an answer; today is about going astray before any material is retrieved.

Source Reading

Hands-On Lab

Confirm three things before starting. One, you have day eight's evaluation script running, since today reuses its 20 questions, metric criteria, and context budget. Two, do not use mean reciprocal rank as evidence: day eight's baseline already has it at 1.0000 because questions written backwards from the corpus put the answer document first — metric saturation rather than a perfect system, and no technique toggled today will move it; single-document recall is likewise already 100%. All of today's evidence must come from multi-hop recall, nDCG, and the refusal rate. Three, read the lab README's MOCK note — offline, all five techniques are deterministic rule-based stand-ins, and the metrics only prove the code path connects rather than representing a real model's effect.

1. Implement query rewriting and reference resolution, run the three-turn conversation, and watch turn two go from can only refuse to the organization chart document entering the context.
2. Implement hypothetical documents and multi-query, fuse the several rankings with day nine's reciprocal rank fusion, and confirm retrievals rise from 20 to 60.
3. Write the intent classifier splitting three ways and give multi-hop its own higher context budget, watching multi-hop recall jump from 50% to 100%.
4. Toggle the five techniques one at a time, recording each configuration's recall, nDCG, model calls, retrievals, and context tokens in one table.
5. Write down your default configuration, explaining line by line why three of them are off — that is today's real assignment.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward choosing among query understanding techniques, reference resolution in multi-turn conversation, and the latency bill of extra calls. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Explain what shortfall each of query rewriting, hypothetical document embeddings, multi-query, and step-back prompting patches
• Implement an intent router that sends chat, retrieval-needed questions, and multi-hop questions down different handling paths
• Prove the net benefit of each query-side technique with evaluation, and work out the extra latency and call count each one adds
• Explain why rewriting comes before routing, and why mean reciprocal rank cannot be evidence today
• All 5 acceptance criteria of the lab pass, with a default configuration and the reasons for switching several off written down
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D11) returns to the index side for parent-child documents, summary indexes, and contextual retrieval. Why that order: today proved that with the retriever unchanged, editing only the query takes multi-hop recall from 50% to 100% — earn the query side's cheap money first, and only then is it the index structure's turn. Tomorrow handles precisely what the query side cannot solve: chunks that lost their locating information when cut, which no amount of query editing restores.