Hybrid search, reranking, and reciprocal rank fusion

Why this matters

Pure vector search misses exact identifiers ("ORD-2026-7842"). Pure keyword search (BM25) misses synonyms. Production RAG runs both, then fuses results, then reranks with a cross-encoder. This three-stage pipeline is the single biggest "I can ship RAG" signal hiring managers grade.

Learning objectives

1. Run BM25 keyword search alongside vector search.
2. Fuse two ranked lists with Reciprocal Rank Fusion (RRF).
3. Rerank candidates with Cohere rerank-3 and open-source bge-reranker-v2-m3.
4. Build the full retrieve → fuse → rerank → answer pipeline.

1. Why hybrid is mandatory

Imagine a user types:

"How to fix ERR_404_X in the payment service"

You have two search systems and they see this query very differently.

Vector search (dense) looks for meaning. It understands that:

• "fix" ≈ "resolve"
• "payment service" ≈ "billing system"

…but it may completely miss the literal token ERR_404_X — that string carries little semantic signal because the embedding model probably never saw it during training.

BM25 (sparse) looks for exact words. It is excellent at finding:

• Error codes (ERR_404_X)
• Product IDs and part numbers
• Legal clauses
• Code identifiers

…but it can completely miss documents that describe the same problem using different wording ("Payment failure 404 troubleshooting guide").

Use only vector → you miss exact codes. Use only BM25 → you miss paraphrases. Use both → you cover the long tail of how real users phrase queries.

Where each method wins

A single retrieval method always loses on roughly half the queries. Fuse both and you reduce failed retrievals by 30-50%.

The score-comparison problem

You cannot just add the two systems' scores together. Vector search returns a cosine similarity around 0.85; BM25 returns a raw term-frequency score around 24.5. These numbers live in completely different universes — they have different ranges, different distributions, and different meanings. Naively summing them lets one side dominate by accident.

This is exactly the problem Reciprocal Rank Fusion solves.

2. BM25 in Python

bm25s is the modern, fast (Rust-backed) BM25 library. Replaces older rank_bm25.

# uv add bm25s
import bm25s

corpus = ["doc one about cats", "doc two about dogs", "doc three about birds"]
retriever = bm25s.BM25()
retriever.index(bm25s.tokenize(corpus))

results, scores = retriever.retrieve(bm25s.tokenize(["dogs and birds"]), k=2)
print(results, scores)

For larger / persistent indexes, use OpenSearch or Elasticsearch (their own BM25 implementation), or Tantivy (Rust, like-Lucene, very fast).

LangChain wraps both nicely:

from langchain_community.retrievers import BM25Retriever
bm25 = BM25Retriever.from_texts(corpus, k=10)
docs = bm25.invoke("dogs and birds")

3. Reciprocal Rank Fusion (RRF)

RRF is the standard way to merge two (or more) ranked lists. It's dead simple, parameter-light, and surprisingly hard to beat — Anthropic, Microsoft, OpenSearch, Elastic, and most production RAG systems use it as the default fusion strategy.

The big idea: ignore scores, use ranks

RRF throws away the raw scores entirely and only looks at the rank position of each document in each list. It asks one question:

"Did this document appear near the top in multiple search methods?"

If yes → boost it. If a document is #1 in one list and #2 in another, it almost certainly belongs at the top of the fused list, regardless of whether the underlying scores are cosine similarities, BM25 scores, or something else entirely.

The formula

For a document d and a set of ranked lists R, the RRF score is:

[ \text{RRF_Score}(d) = \sum_{m \in R} \frac{1}{k + r_m(d)} ]

Where:

• r_m(d) is the rank (1-indexed) of document d in list m. If d doesn't appear in list m, that term is 0.
• k is a smoothing constant (almost always 60, from the original Cormack et al. 2009 paper). Larger k dampens the gap between top ranks; smaller k over-rewards being #1.

Documents that appear near the top of both lists climb fastest because their reciprocal-rank contributions stack.

The judges analogy

Imagine two judges ranking students for a scholarship:

Who wins overall? You can't compare scores — Judge A might grade out of 100, Judge B out of 10. So you compare ranks:

• Alice appears at #1 (A) and #2 (B) → ranked highly by both.
• Bob appears at #2 (A) and #1 (B) → ranked highly by both.
• Charlie appears at #3 in only one list.
• David appears at #3 in only one list.

Alice and Bob rise to the top. That's exactly what RRF does with documents.

Worked example

Query: "How to fix ERR_404_X in payment service"

Vector search top-3:

BM25 top-3:

With k = 60:

D3 wins (top of BM25, also retrieved by vector). D1 is a close second (top of vector, also retrieved by BM25). The documents only one method liked (D2, D4) sit clearly below.

This is the magic: agreement between methods is automatically rewarded, and you didn't have to normalise or tune anything except k.

Implementation

from collections import defaultdict

def rrf(lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """
    Fuse multiple ranked lists into one.

    lists: each item is an ordered list of doc_ids, best first.
    k:     smoothing constant (60 is the canonical default).

    Returns: [(doc_id, fused_score), ...] sorted high to low.
    """
    scores: dict[str, float] = defaultdict(float)
    for ranked in lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Why k = 60? It dampens the gap between rank 1 and rank 2 (1/61 vs 1/62) so a single judge's #1 pick can't single-handedly dominate. The Cormack paper showed 60 is near-optimal across many TREC datasets; in 16 years nobody has found a meaningfully better default.

Alternative: weighted score fusion

When you do trust your raw scores and have a fixed schema, you can normalise both to [0, 1] and combine:

def weighted_fusion(vector_top: list[tuple[str, float]],
                    bm25_top: list[tuple[str, float]],
                    w_vec: float = 0.7, w_bm25: float = 0.3) -> list[tuple[str, float]]:
    out = defaultdict(float)
    for did, s in vector_top: out[did] += w_vec * s
    for did, s in bm25_top:   out[did] += w_bm25 * s
    return sorted(out.items(), key=lambda x: x[1], reverse=True)

Anthropic's Contextual Retrieval cookbook uses 0.8 / 0.2 (semantic / BM25). Tune the weights on your eval set. Always normalise raw scores first — mixing un-normalised cosine and BM25 scores is the #1 bug in homegrown hybrid systems.

When to pick RRF vs weighted fusion

• RRF — your default. Zero tuning, robust across datasets, easy to add a third retriever (e.g. ColBERT) later.
• Weighted fusion — only when you have a held-out eval set and want to squeeze out the last few % of recall, and you've already normalised scores cleanly.

One-sentence summary

Hybrid Search with RRF solves the problem of missing either exact keywords (BM25's strength) or semantic meaning (vector search's strength) by running both retrievers in parallel and ranking documents by how highly they appear in each result list, rather than trying to compare incompatible scores.

4. Reranking — the cheap 30% boost

Vector search gives you 50 candidates. Most are noise. A cross-encoder reranker scores each (query, candidate) pair using full attention — much more accurate than embedding cosine — but slower, so we only run it on the top-50.

With Cohere (managed, dead simple)

# uv add cohere
import cohere
co = cohere.ClientV2()  # reads COHERE_API_KEY

response = co.rerank(
    model="rerank-3",   # 2026 default
    query="how do I reset my password",
    documents=[c["text"] for c in candidates],
    top_n=5,
)
ranked = [candidates[r.index] for r in response.results]

Cohere rerank-3 is multilingual, supports up to 4k tokens per doc, ~50ms latency, ~$2/1k searches.

With open-source bge-reranker-v2-m3

# uv add sentence-transformers
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", trust_remote_code=True)
pairs = [(query, c["text"]) for c in candidates]
scores = reranker.predict(pairs)
ranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)][:5]

Free, local, GPU-friendly. Slightly worse than Cohere but very competitive.

When to rerank

Always, for production. The cost is small (you rerank ~50 docs, one cross-encoder call). The lift is large (typically +10-25% recall@5).

5. The full pipeline

def retrieve(query: str, k_final: int = 5) -> list[dict]:
    # 1. Dense
    dense_hits = collection.query(query_texts=[query], n_results=50)["ids"][0]

    # 2. BM25
    bm25_hits = bm25_retriever.invoke(query)         # 50 ids
    bm25_ids = [d.metadata["chunk_id"] for d in bm25_hits]

    # 3. Fuse with RRF
    fused = rrf([dense_hits, bm25_ids], k=60)         # [(id, score), ...]
    top50_ids = [did for did, _ in fused[:50]]

    # 4. Lookup texts
    candidates = lookup_texts_by_ids(top50_ids)

    # 5. Rerank with Cohere
    rr = co.rerank(model="rerank-3", query=query,
                   documents=[c["text"] for c in candidates],
                   top_n=k_final)
    return [candidates[r.index] for r in rr.results]

That is the production pattern. Memorise it.

6. Putting it into your RAG flow

SYSTEM = """You answer using only the provided sources.
If the sources are insufficient, say so. Cite sources by their [n]."""

def answer(query: str) -> str:
    sources = retrieve(query, k_final=5)
    bullets = "\n\n".join(f"[{i+1}] {s['text']}" for i, s in enumerate(sources))
    r = client.responses.create(
        model="gpt-4.1-mini",
        instructions=SYSTEM,
        input=f"QUESTION: {query}\n\nSOURCES:\n{bullets}",
    )
    return r.output_text

Now you have a defensible pipeline:

• dense → semantic recall
• BM25 → exact-match recall
• RRF → fusion
• rerank → precision
• strict prompt → grounded answer

7. Latency budgets (real numbers from production)

For a smooth chat UI you want sub-2-second total. A rough budget:

Cache embeddings of common queries with Redis to save the 30-60ms.

Hands-on lab (4 hours)

Build hybrid_rag.py:

1. Reuse the chunked corpus from Lesson 2.2.
2. Build a Chroma collection (vector) and a bm25s index.
3. Implement rrf() and a retrieve() function with both stages.
4. Add reranking with Cohere rerank-3. (Free tier: 1k requests/month.)
5. On your 30-question eval set, measure recall@5 for: vector-only, BM25-only, hybrid (RRF only), hybrid + rerank.
6. Print a markdown table.
7. Add a small Streamlit UI that shows the top sources and the answer.

Acceptance criteria:

• Hybrid + rerank beats vector-only by ≥ 10% recall@5.
• Latency p95 < 1.5s on a laptop with k_final=5.
• README explains why each stage matters.

Common pitfalls

1. Different ID spaces between BM25 and vector store. Use a shared chunk_id everywhere.
2. Re-embedding the rerank input. Reranker takes raw text, not vectors.
3. Reranking too many candidates. Cap at 50, it is enough.
4. Forgetting RRF's k — too low (k=10) over-weights top-of-list; 60 is standard.
5. Mixing units in weighted fusion. Normalise scores to [0,1] first.

Self-check

1. Why does cosine ranking diverge from BM25 ranking on rare-token queries?
2. What does the k=60 constant in RRF do?
3. Why is a cross-encoder slower than dual-encoder embeddings?
4. When would you skip BM25?
5. How do you measure rerank quality independently of retrieval?

References

• Anthropic Contextual Retrieval (RRF + BM25 + rerank example)
• bm25s GitHub (fast BM25)
• Cohere rerank-3
• BAAI bge-reranker-v2-m3
• LangChain ensemble retriever
• RRF paper: Cormack et al. 2009 (https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf)