Hybrid Search and Reranking: Two-Path Retrieval, Reciprocal Rank Fusion, Then Re-Ranking the Top Results With a Cross-Encoder

Scores cannot simply be added: two rulers with different graduations

The most natural idea is adding the two paths' scores and sorting. That idea shatters on contact.

A BM25 score is a sum of logarithmic terms with no upper bound. On this corpus, the golden set's 20 questions have first-place scores from 16.37 to 94.46, and a more obscure phrasing goes as low as 6.49 — depending on how rare your terms are and how often they hit. Cosine similarity is pinned between minus one and one, and under MOCK=1 hash vectors even a relevant chunk sits just past 0.2. One is a rubber ruler with no graduations and the other a vernier caliper graduated too finely to read, and adding the two readings means nothing.

What about normalizing first — divide each by its own path's top score, squeeze into 0 to 1, and weight? Week one's system (day 7's implementation) did exactly that; it works, and it has a silent defect: the denominator floats with the query. Suppose a question has no answer in the corpus at all, so the vector path fetches only irrelevant chunks with a top score of 0.09 — after normalization that least relevant result scores a full 1.0 and marches into the fusion with its weight. You believe you are comparing how relevant, and you are comparing how tall among this path's dwarves.

The weights themselves are worse. Where did 1 : 0.6 come from? Day 7 guessed it. Tuning it means running a full evaluation per weight combination — two paths are two-dimensional, three are three-dimensional, and adding query rewriting's multi-path recall (day 10) makes four or five. That is a tuning problem expanding exponentially with the number of paths, and it must be redone from scratch with every embedding model change.

The conclusion is blunt: scores are incomparable and ranks are not. Whether BM25 gave 52 or 5, the sentence "it ranked first on the keyword path" has stable meaning.

Reciprocal rank fusion: only ranks, and one constant is enough

Reciprocal rank fusion (RRF) is almost unreasonably simple: each path gives an ordered list, a document ranked i on a path scores 1 / (k + i) there, the paths' scores are summed, and the highest ranks first. k is a constant, uniformly 60 across this course.

export const RRF_K = 60
 
export function rrfFuse(rankings, k = RRF_K) {
  const scores = new Map()
  for (const ranking of rankings) {
    ranking.forEach((id, index) => {
      // Ranks start at 1
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + index + 1))
    })
  }
  return [...scores.entries()]
    .map(([id, score]) => ({ id, score }))
    .sort((a, b) => b.score - a.score || (a.id < b.id ? -1 : 1))
}

RRF_K = 60
 
 
def rrf_fuse(rankings: list[list[str]], k: int = RRF_K) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for index, doc_id in enumerate(ranking):
            # Ranks start at 1
            scores[doc_id] = scores.get(doc_id, 0.0) + 1 / (k + index + 1)
    # Break ties by id so repeated runs give identical results
    return sorted(scores.items(), key=lambda item: (-item[1], item[0]))

Note the signature: the input is each path's ordered list of ids, not a list of scores. The scores were discarded before entering this function, which is not laziness but design — discarding what is incomparable prevents being misled by it.

k acts as a flattener. At k of 0, first place scores 1.0 and second 0.5, a factor of two, meaning whoever ranks first on their own path wins; at k of 60, first scores 0.0164 and second 0.0161, a difference of 1.6%, so ranking near the top on both paths carries more weight than ranking first on one. That is exactly what we want: a suspect cross-validated by both detectives goes first. The lab's fifth section has you sweep k from 0 to 600 and watch one question's top 5 deform accordingly.

Setting the recall depth: this knob is harsher than you think

Before fusing, how many does each path take? This course uniformly takes the top 50, keeping the top 20 after fusion for reranking.

But that default has a precondition. In a million-chunk corpus, 50 is one ten-thousandth, conservative to the point of harmlessness; and this lab's corpus is 134 chunks, so 50 per path pours 37% of the store into the fusion — rather like having two detectives list half the city as suspects. Noise has ranks too, and being on the list earns votes.

The lab's sixth section runs that into a table (30 documents, 134 chunks, D8's 20 questions, a 600-token context budget):

per path   recall   multi    nDCG@10   refusal
5          93.8%    75.0%    0.6281    0.0%
10         87.5%    50.0%    0.6826    0.0%
20         87.5%    50.0%    0.7054    0.0%
50         87.5%    50.0%    0.7186    0.0%

Two metrics openly fight here: pull recall depth back to 5 and both recall and the multi-hop tier return while nDCG@10 falls by nearly 0.1. The reason is not hard — take fewer and less garbage enters the candidates, so the answer squeezes into the 600-token budget more readily; take more and the top 10's overall quality looks better while the answer gets pushed out of the context by more chunks that also look relevant.

That is why how many to take has no standard answer and only an evaluation result. If your business is question answering, recall comes first; if it is a search results list for a human, ranking quality does. Decide which you want and then read the table.

Bi-encoders and cross-encoders: one precomputed, one computed on the spot

There is one more layer after fusion. Everything up to here computed similarity with a bi-encoder: the query encodes to a vector, the document encodes to a vector, and the two take one dot product. The key is that they never met — document vectors can be computed offline, indexed, and searched across the whole store in milliseconds. That is the entire reason it withstands massive data.

The cost is information lost in compression. A 500-word document squeezed into 1,536 numbers may not retain the binding between a person's name and "team lead" in the sentence stating it.

A cross-encoder changes the structure: query and document are concatenated into one text and sent through the model together, with every attention layer letting the query's words look at the document's. The question "who is the lead" can align directly onto the name. It is far more accurate — at the cost that nothing can be precomputed. N documents mean N forward passes, each waiting on the model.

So it can only stand behind recall as a reranking layer: recall fetches a small batch from the whole store, and reranking fixes that batch's order. In the library metaphor, recall is carrying an armful of books to the desk by the card catalog, and reranking is the librarian opening each one to pick the three you should read — they cannot open the whole collection.

const RRF_K = 60, PER_ROUTE_K = 50, FUSE_KEEP = 20
 
export async function hybridSearch(query, routes, rerank) {
  // Two paths in parallel: the keyword path is pure in-memory computation, the vector path awaits one embedding
  const [keyword, vector] = await Promise.all([
    routes.bm25(query, PER_ROUTE_K),
    routes.vector(query, PER_ROUTE_K),
  ])
  const fused = rrfFuse([keyword.map((h) => h.id), vector.map((h) => h.id)], RRF_K)
  // Only the top 20 go to the cross-encoder, not the whole store
  const candidates = fused.slice(0, FUSE_KEEP).map((f) => ({ id: f.id, text: routes.text(f.id) }))
  return rerank(query, candidates)
}

RRF_K, PER_ROUTE_K, FUSE_KEEP = 60, 50, 20
 
 
async def hybrid_search(query, routes, rerank):
    # Two paths in parallel: the keyword path is pure in-memory computation, the vector path awaits an embedding
    keyword, vector = await asyncio.gather(
        routes.bm25(query, PER_ROUTE_K),
        routes.vector(query, PER_ROUTE_K),
    )
    fused = rrf_fuse([[h.id for h in keyword], [h.id for h in vector]], RRF_K)
    # Only the top 20 go to the cross-encoder, not the whole store
    candidates = [{"id": doc_id, "text": routes.text(doc_id)} for doc_id, _ in fused[:FUSE_KEEP]]
    return await rerank(query, candidates)

Reranking's cost accounting: three numbers decide whether to adopt it

Reranking is this course's first optimization that explicitly costs money and explicitly costs time, so it must clear three bills — metric, latency, cost — plus a fourth that is often forgotten.

Bill one, the metric — but first confirm which metric you are reading. Yesterday's baseline already had mean reciprocal rank at 1.0000, as do three of today's four configurations. That is not full marks but metric saturation: questions written backwards from the corpus with such high literal overlap always put the answer document first, so it cannot rise further. When a metric hits the ceiling the response is writing harder questions, not declaring victory. So today's evidence can come only from recall (especially the multi-hop tier), nDCG, and the refusal rate, the three with room left.

By those three, reranking's contribution attributes precisely to one question. q07 requires hitting both doc-029 and doc-021: the pure keyword tier hits it, fusion loses it (doc-021 diluted and pushed out of the 600-token budget), and reranking brings it back — taking the multi-hop tier from 50.0% back to 75.0%. That is a clean, attributable improvement: not "hybrid search raised the metric" but "reranking fixed the one question fusion broke." As for q06, also multi-hop, all four tiers miss it, because the answer document never entered the candidate pool at all — fusion and reranking can only reorder existing candidates and cannot conjure new ones, and that debt is paid later.

The refusal rate is another matter entirely: reranking helps not at all in this implementation. Admission is gated on each path's raw score, and reranking changes order rather than admission; the only tier raising the refusal rate is pure vector (25.0%, with the 0.2 cosine gate genuinely blocking one unanswerable question). Reranking cannot rescue failing to stay quiet when it should, which is day 6's generation-side refusal work.

Bill two, latency. Reranking is one synchronous network round trip wedged between retrieval and generation, with the user waiting throughout. The lab's column uses overridable placeholder values: 120 milliseconds per embedding and 180 per rerank. So the pure BM25 tier's modeled latency is near zero, hybrid is 120 milliseconds, and hybrid plus reranking is 300. Those 180 milliseconds buy ranking quality, not answer quality — if your context budget is large enough that the answer got in anyway, that money was wasted.

Bill three, money. Reranking is generally priced per search rather than per token: one question counts as one search whether it sends 20 candidates or 100. That explains two things — why reranking's cost is per question rather than per chunk, and why sending a few more candidates barely raises the fee, since what is genuinely expensive is the number of questions. The lab's reranking input for that tier is 2,922 tokens per question across 1 call, while the embedding path is only 31.6 tokens per question. Two orders of magnitude apart, meaning the retrieval side's bill is essentially the reranking bill.

Bill four, whether it can be self-hosted. This one is often forgotten and is frequently decisive. A cross-encoder is not as vast as a large model: the lab's default hosted backend is Cohere's rerank-v3.5, and the self-hosted path uses BAAI/bge-reranker-v2-m3 — Apache-2.0, multilingual, and fitting on one consumer graphics card. When data may not leave the network, or the volume makes per-call pricing uneconomical, switch to self-hosting; rerank.ts's two backends implement one interface and it is one environment variable.

Metadata filtering before or after retrieval

The last piece. Real retrieval is rarely across the whole store and is usually within this department, this time range, or what this person may see. Where the filter goes depends on how much remains after it.

Filter first, then retrieve (pre-filtering): do this when the range is small. With 200 documents left, running BM25 and vectors over those 200 is fast, and all 50 recalled are legitimate.

Retrieve first, then filter (post-filtering): do this when the range is large. But it has a pothole to guard: recall 50, filter out 45, and you hold 5 — quietly cutting the recall depth by 90%. The remedy is scaling the recall count by the pass rate: an estimated 20% pass rate means recalling 250 and then filtering.

// Estimate what this filter leaves before deciding which step it belongs to
export function planFilter(estimatedKept, totalDocs, want = 50) {
  const ratio = estimatedKept / totalDocs
  // Under a tenth remaining, narrowing first is better value: a small candidate pool runs both paths faster
  if (ratio < 0.1) return { mode: 'pre', topK: want }
  // Otherwise post-filter, scaling the recall count by the pass rate, or little is left afterwards
  return { mode: 'post', topK: Math.ceil(want / Math.max(ratio, 0.05)) }
}

def plan_filter(estimated_kept: int, total_docs: int, want: int = 50) -> dict:
    ratio = estimated_kept / total_docs
    # Under a tenth remaining, narrowing first is better value
    if ratio < 0.1:
        return {"mode": "pre", "top_k": want}
    # Otherwise post-filter, scaling the recall count by the pass rate
    return {"mode": "post", "top_k": -(-want // max(ratio, 0.05))}

Permission filtering is the deadliest case of this, because one mistake is an incident — and that whole subject waits for day 13. Here you only need the criterion: compute the pass rate first, then decide the order.

Source Reading

Hands-On Lab

starter/ has 4 exercises cut out: reciprocal rank fusion, the k sweep, the rerank stand-in's scoring, and the cost ledger. Run as-is, all four acceptance items are ❌ and each one finished turns one green — use it as a progress bar and do not read the answers all at once. None of the four needs a network or an API key.

1. Run the starter as-is first, note the four-tier table's numbers, and observe that the hybrid tier is currently identical to pure BM25 — because fusion is not implemented.
2. Implement rrfFuse and sweepRrfK, run again: section five's k sweep table appears, showing how the top 5 deforms as k goes from 0 to 600.
3. Implement the rerank stand-in's scoring, and section seven shows how many positions were rewritten; compare nDCG before and after to judge whether it helped or hurt.
4. Implement the cost ledger and section three's cost column stops being 0; set PRICE_RERANK_PER_KSEARCH to a tenth and rerun to see how the four tiers' cost shares invert.
5. Rerun with a few different recall depths via PER_ROUTE_K, watch the scene where recall and nDCG fight, then write down your recommended configuration and why.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward fusion strategies for multi-path recall, a reranking model's cost and benefit, and metric-driven trade-offs. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Explain the blind spots of keyword search and vector search respectively, and give a query only one of them can hit
• Implement Reciprocal Rank Fusion and explain why it's more stable than weighting by raw score
• Explain the structural difference between bi-encoders and cross-encoders, and why reranking can only be applied to the top few dozen results
• Say what happens when ties are not broken deterministically
• All 5 acceptance criteria of the lab pass
• Answer at least 3 of the 4 interview questions without looking at the key points

Today's effort went into what happens after retrieval: how two paths fuse and how the fusion is refined. But q06 already drew this road's ceiling — all four configurations fail, because no path ever fetched the answer document into the candidate pool. With the exit side largely wrung out, tomorrow (D10) moves to the entrance: a user's question is short and vague, and failing to retrieve is often not the index's fault. Rewriting, hypothetical document embeddings, multi-query, step-back prompting, and intent routing — ask the question properly before discussing how to retrieve. Incidentally, the several paths multi-query generates are fused by today's rrfFuse with not one line changed.

If you took the 30-day course, day 24 covers what this looks like once wired into an agent, and is worth reading alongside.