Hybrid Search & Reciprocal Rank Fusion (RRF).

🧩The problem you hit around week 3 of RAG

Your vector search is good at "how do I cancel my plan" → finds the doc titled "Ending a subscription". Semantics work.

Then a user searches "ERR_TLS_1042" or "flag --no-verify-peer" and you get back three friendly articles about security in general. Zero exact hits.

Embeddings compress meaning. Rare tokens — error codes, SKUs, CLI flags, surnames — are exactly what gets compressed away. BM25 (plain keyword/lexical search) nails those, because a rare term has huge IDF weight.

So: run both, then fuse. That's hybrid search.

⚙️Build it in 5 lines (Reciprocal Rank Fusion)

def rrf(rankings, k=60):
    scores = defaultdict(float)
    for ranked_ids in rankings:              # 1
        for rank, doc_id in enumerate(ranked_ids, start=1):   # 2
            scores[doc_id] += 1.0 / (k + rank)                # 3
    return sorted(scores, key=scores.get, reverse=True)       # 4

top = rrf([bm25_search(q, 50), vector_search(q, 50)])[:10]    # 5

What each line teaches:

1️⃣ Input is lists of IDs, not scores. Deliberate.

2️⃣ Only rank position matters — position 1, 2, 3…

3️⃣ The whole algorithm. Contribution decays as 1/(k+rank).

4️⃣ A doc in both lists sums both contributions → agreement wins.

5️⃣ Over-fetch (50) from each side, return 10. Fusion needs candidates to fuse.

🔑Why ranks, not scores — the bit people get wrong

BM25 returns unbounded scores (0 → 40+, corpus-dependent). Cosine similarity returns ~0.0–1.0, and in practice clusters tight: your best hit is 0.86, your worst is 0.81. Those two scales have no common unit.

The usual fix — min-max normalize each list, then 0.7*dense + 0.3*bm25 — is fragile: normalization depends on the min/max in that result set, so one outlier rescales everything and your weights silently change per query. RRF sidesteps it by throwing scores away entirely.

🔢What k=60 actually does

k damps the top of the list. With k=60:

• rank 1 → 1/61 = 0.0164
• rank 10 → 1/70 = 0.0143 (only 13% less)
• rank 50 → 1/110 = 0.0091

So rank 1 is NOT dominant — a doc at rank 8 in both lists (0.0294) beats a doc at rank 1 in one list and absent from the other (0.0164). That's the intended behavior: cross-retriever agreement is the signal.

Lower k (say 10) makes it top-heavy: rank 1 = 0.091 vs rank 10 = 0.05. Tune k, don't treat 60 as sacred — it's just the value from the original 2009 paper.

⚠️Three traps

• Fetching only top-10 per retriever. A doc at BM25 rank 23 never gets to redeem a weak dense rank. Over-fetch 5x what you return.
• Assuming RRF replaces a reranker. It doesn't. RRF = cheap candidate merging. A cross-encoder reranker on the fused top-50 is still where most quality comes from.
• Feeding it three retrievers where two are near-duplicates (e.g. two embedding models from the same family). They vote together and drown the lexical signal. Diversity of failure modes is the point.

📊 Rough expectation: on BEIR-style benchmarks, hybrid typically buys +5–12 points nDCG@10 over dense alone, and the gain is biggest on entity/code/ID-heavy corpora — smallest on chatty FAQ text.

🏋️Your exercise

Take 10 real queries from your app. Split them: 5 "conceptual" (how do I…), 5 "literal" (an error code, a function name, a product ID).

Run each through dense-only and BM25-only, record the rank of the correct doc in both.

Then compute RRF by hand for the worst literal query. Question to answer: did fusion actually pull the right doc up, or was it already absent from both top-50s? If it's absent, your problem is chunking or recall — not fusion.