BM25 vs embeddings vs rerankers: choosing search stages
BM25 ranks text by matching query terms. Embeddings find candidates with similar meaning. A reranker scores the retrieved candidates again to improve their order. For engineers building search or RAG, these stages address different failures and should be tested separately.
Start with BM25 and enforce document permissions. Add embeddings when relevant passages are missing because the query uses different words. Add a reranker when the right passages are retrieved but appear too low in the list. Keep each stage only if it improves labeled results within the time and cost budget.
For example, a query for ERR-42 needs correct identifier handling. A question about failure after wake may need semantic matching if the manual says “resume failure”. Reranking helps only after the required passage enters the candidate set.
What each stage does
| Stage | Method | Job |
|---|---|---|
| Filtering | Structured filters | Enforce tenant, permissions, product, language, time, and availability. |
| Lexical retrieval | BM25 | Matching terms; preserve IDs and error codes with suitable fields or analyzers. |
| Dense retrieval | Embeddings | Synonyms, paraphrases, fuzzy intent, and semantic recall. |
| Fusion | Reciprocal Rank Fusion or weighted retriever composition | Merge sparse and dense candidates without pretending scores are comparable. |
| Reranking | Cross-encoder | Read the query together with each candidate to score a shortlist. |
| Final precision | LLM reranker or answer model | Resolve nuanced relevance only after the list is small. |
| Evaluation | Recall@k, nDCG, MRR, click labels, human labels | Check whether each added stage improves the baseline. |
Use-case defaults
| Product | Approach to test | Why |
|---|---|---|
| Documentation search | BM25 plus embeddings plus cross-encoder | Exact API names and semantic questions both matter. |
| RAG retrieval | BM25 baseline; compare hybrid retrieval and reranking | Score evidence coverage, answer support, and latency. |
| Product search | Lexical filters plus hybrid retrieval plus business features | Availability, price, popularity, and exact facets matter. |
| Support search | Hybrid retrieval plus freshness and ticket metadata | Similar wording and current policy both matter. |
| Internal knowledge base | BM25 baseline, then dense retrieval from query logs | Start measurable before adding model cost. |
| Legal or compliance search | Lexical baseline plus strict filters, then careful semantic expansion | False positives and false negatives both have high cost. |
Why BM25 still belongs in the stack
Embeddings find text with similar meaning, but they do not reliably replace exact matching. Error codes, function names, product SKUs, legal phrases, and people’s names often carry intent through their exact spelling. BM25 is a useful baseline because it rewards matching terms. Text analysis may change those terms: the Elasticsearch standard analyzer splits text and lowercases it. Preserve punctuation-sensitive IDs such as ERR-42 with a suitable analyzer or an exact-value field. BM25 ranking does not guarantee exact string equality.
Dense retrieval can add relevant results when users do not know the document’s vocabulary. Test it alongside BM25. If the two retrievers find useful complementary results, RRF can combine their ranked lists. It uses result positions rather than comparing raw lexical and embedding scores.
When to add a reranker
Add a cross-encoder when relevant documents enter the candidate set but rank too low. A cross-encoder reads the query together with each candidate document. That requires a model pass for each pair, so apply it to a shortlist. Choose the candidate count from measured recall and latency; top 50 is a useful experiment, not a universal threshold.
Test an LLM reranker when a cross-encoder misses distinctions your application needs, such as which version of a policy applies. It adds model work whose cost depends on the candidate count and input length. Compare quality, latency, and cost on the same shortlist before keeping it.
Recall@k measures how much labeled relevant material appears in the first k results. MRR rewards a high position for the first relevant result; nDCG measures ordering when labels have different relevance levels. Precision@1 asks whether the first result is relevant.
Evaluation sequence
- Label real queries across important intents, languages, permissions, and failure costs. Start small, then expand until the slices and uncertainty support the decision.
- Measure BM25 alone.
- Test dense retrieval when BM25 misses relevant passages.
- If the two result sets complement each other, test fusion with nDCG and Recall@k.
- Test cross-encoder reranking when relevant candidates rank too low; measure Precision@1 and nDCG.
- Add LLM reranking only if it improves quality after cost and latency are included.
- Watch production metrics: zero-result rate, reformulation rate, click-through, answer correction, p95 latency, and cost.
Deeper reading
- Search Ranking Stack gives the full implementation walkthrough.
- RAG Evaluation Metrics explains retrieval and citation evaluation.