Reranking (Cross-Encoders in RAG).

🐞The symptom

Your RAG bot answers a question wrong. You check the vector DB — the correct passage IS in there. You even watch it come back from search... at rank #7. But you only feed the top 3 chunks to the LLM, so the model never sees the answer. Classic retrieval-quality bug, and it's invisible unless you print the ranks.

🔍Why vector search alone does this

Embedding search uses a bi-encoder: it turns the query into ONE vector and each document into ONE vector (precomputed), then compares by cosine similarity. Blazing fast — millions of docs in milliseconds — but lossy. The whole meaning of a passage is squished into a single ~1024-dim vector before it ever meets your query. Subtle relevance ("does THIS passage actually answer THIS question?") gets blurred, so the truly-best chunk often lands at rank 5–20 instead of 1–3.

🔧The fix: a two-stage retrieve → rerank pipeline

1. Retrieve wide — pull top 50 with fast vector search (the recall stage).
2. Rerank — run a cross-encoder over those 50. A cross-encoder takes (query, passage) TOGETHER as one input and outputs a single relevance score. Far more accurate because it reads them jointly, but slow — so you only run it on 50 candidates, never the whole corpus.
3. Keep the top 3 reranked. Now the right chunk sits at #1.

🧩In ~5 lines:

hits   = vector_db.search(query, top_k=50)          # cheap, high recall
scored = reranker.rank(query, [h.text for h in hits])
top    = [hits[i] for i in scored.top(3)]           # precise
answer = llm(build_prompt(query, top))

🧠Mental model

Bi-encoder = "index everyone's résumé once, then grab the 50 closest to the job." Cross-encoder = "actually interview those 50 for THIS job." You can't interview a million candidates — but you can interview 50. Wide-then-narrow.

⚖️The tradeoff to remember

Recall vs precision. Vector search maximizes recall cheaply; reranking maximizes precision expensively. Chaining them gives you both. Retrieving top_k=3 directly and skipping the rerank is the #1 reason "the answer is literally in my DB but the bot can't find it."

🎯Your exercise

Take a RAG query that's answering poorly. Bump retrieval to top_k=50 and print the rank of the gold passage — is it sitting at rank 8? Add a cross-encoder rerank step, keep the top 3, and re-ask. Where did the gold chunk move to, and did the answer flip to correct?