Cross-Encoder Re-Ranking & Two-Stage Retrieval
Transform raw vector retrieval into precision relevance: Combine high-speed Bi-Encoder candidate generation with full-attention Cross-Encoder re-ranking.
1. Why Standard Vector Search Falls Short Alone
In production RAG, retrieving the top-5 documents directly from a vector database and passing them to an LLM frequently results in subtle hallucinations. Why? Because vector databases use Bi-Encoders, which compress text into isolated vectors before seeing the user's query.
The Production Two-Stage Funnel
- Stage 1: Candidate Generation (Bi-Encoder / Vector DB): Fast, approximate search over millions of documents in 3ms to 10ms. Returns top 50 to 100 candidate chunks with high recall.
- Stage 2: Precision Re-Ranking (Cross-Encoder): Deep neural re-scoring of the 50 candidate chunks in 20ms to 40ms. Cross-attention evaluates the query and document simultaneously, sorting the top3 to 5 highest-relevance passages for the LLM context prompt.
2. Bi-Encoder vs. Cross-Encoder: Attention Mechanics
| Property | Bi-Encoder (Vector Databases) | Cross-Encoder (Re-Rankers) |
|---|---|---|
| Input Encoding | Query q and Doc d encoded independently: u = f(q), v = f(d) | Concatenated joint input: [CLS] query [SEP] document [SEP] |
| Cross-Token Attention | None. Query tokens never interact with document tokens. | Full Cross-Attention. Every query token attends to every document token across all layers. |
| Precomputability | Yes. Billions of document vectors can be indexed in advance. | No. Must be evaluated live at query time for every candidate pair. |
| Accuracy / Relevance | Moderate to High | State-of-the-Art Precision |
| Throughput Scalability | Scales to 100M+ documents via ANN graphs | Limited to ~100 candidate documents per query |
3. Production Re-Ranker Landscape
Cohere Rerank v3
Cloud managed API (rerank-english-v3.0, rerank-multilingual-v3.0). Zero local GPU required. Industry standard for multilingual enterprise RAG.
BAAI BGE-Reranker
Leading open-source HuggingFace model (BAAI/bge-reranker-large, bge-reranker-v2-m3). Can be self-hosted on Nvidia T4/A10G GPUs via vLLM or HuggingFace TEI.
FlashRank
Ultra-lightweight Python library. Uses quantized 4MB ONNX models running directly on CPU with sub-10ms latency. No PyTorch or GPU required!
4. Hands-On Python: Re-Ranking Candidates with SentenceTransformers
Here is a practical Python script demonstrating how a Cross-Encoder re-scores candidates retrieved from an initial vector search:
from sentence_transformers import CrossEncoder
# Load a high-performance cross-encoder model
cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
query = "What is the capital requirement for Tier 1 banks under Basel III?"
# Simulate 4 candidate documents retrieved from Vector DB (Bi-Encoder)
candidate_passages = [
"Basel III requires banks to maintain a minimum common equity Tier 1 (CET1) ratio of 4.5% of risk-weighted assets.",
"Central banks regulate fractional reserve banking across global economies.",
"Investment banking divisions provide underwriting and mergers advisory.",
"Basel III introduced the Liquidity Coverage Ratio (LCR) to withstand short-term liquidity shocks."
]
# Step 1: Format pairs for Cross-Encoder: [[query, doc1], [query, doc2], ...]
pairs = [[query, doc] for doc in candidate_passages]
# Step 2: Compute full cross-attention relevance scores
scores = cross_encoder.predict(pairs)
# Step 3: Sort candidates in descending order of precision relevance
ranked_results = sorted(zip(candidate_passages, scores), key=lambda x: x[1], reverse=True)
print("=== Cross-Encoder Re-Ranked Results ===")
for rank, (doc, score) in enumerate(ranked_results, 1):
print(f"Rank {rank} (Score: {score:.4f}):\n {doc}\n")RAG Evaluation & Metrics
Continue to the next advanced retrieval and vector search engineering module.
Explore Related Tools
Hand-picked utilities and calculators related to this tool.
