Reranking in RAG Pipelines: Cross-Encoders, ColBERT, and

Table of Contents:

• What Is Reranking in RAG?
• Why First-Stage Retrieval Is Not Enough
• Reranker Types: Cross-Encoder vs ColBERT vs LLM
• Building a Two-Stage RAG Pipeline in Python
• Reranker Model Selection and Benchmarks
• Latency, Cost, and Scaling Strategies
• Production Patterns and Anti-Patterns
• Frequently Asked Questions

What Is Reranking in RAG?

Short answer: Reranking in RAG pipelines is a second retrieval stage that rescores first-stage candidates with a more accurate (and slower) model — typically a cross-encoder — to surface the most relevant chunks before LLM generation.

First-stage retrieval casts a wide net: embed the query, pull top-50 candidates via approximate nearest neighbor search. It is fast but imprecise — embeddings compress meaning into single vectors, and ANN indexes sacrifice accuracy for speed. Reranking re-evaluates those 50 candidates with a model that sees the full query and document together, producing much sharper relevance scores.

Key Takeaways:

• Reranking improves precision@5 by 15-25% on typical production corpora
• Use a two-stage pattern: fast bi-encoder retrieval (top-50) → cross-encoder rerank (top-5)
• Cross-encoder reranking adds 50-150ms — acceptable for most RAG applications
• Always measure with retrieval eval before and after adding reranking
• Reranking fixes ranking problems, not recall problems — if the right doc is not in top-50, reranking cannot help

This is Symptom 5 in our guide on why RAG pipelines return garbage. Teams skip reranking because first-stage results "look reasonable" — then the LLM synthesizes an answer from the third-best chunk instead of the first. The same teams often explore RAG vs fine-tuning before fixing retrieval ranking — reranking is faster, cheaper, and addresses the actual bottleneck.

Why First-Stage Retrieval Is Not Enough

Short answer: Bi-encoder retrieval ranks by vector similarity, which correlates with but does not equal relevance — reranking closes the gap by scoring query-document pairs jointly.

The Bi-Encoder Limitation

Bi-encoders embed query and document independently:

score = cosine(embed(query), embed(document))

The model never sees query and document together at scoring time. This causes systematic ranking errors:

What Reranking Fixes

Cross-encoders process query and document together through a transformer:

score = CrossEncoder("[query] [SEP] [document]")

The model attends across both inputs — it understands negation, specificity, and multi-aspect matching. This is why reranking improves precision (top results are more relevant) even when recall (right doc is in the candidate pool) stays the same.

What Reranking Does NOT Fix

If first-stage retrieval misses the correct document entirely, reranking is useless. Before adding a reranker:

1. Run retrieval evaluation — check recall@50
2. If recall@50 < 0.85, fix chunking, embeddings, or add hybrid search
3. If recall@50 ≥ 0.85 but precision@5 < 0.70, add reranking

This diagnostic sequence saves weeks of misdirected optimization on RAG & LLM systems.

Reranker Types: Cross-Encoder vs ColBERT vs LLM

Short answer: Production RAG pipelines use cross-encoder rerankers for the reranking stage in 90% of cases — ColBERT for larger candidate pools, LLM reranking only for high-stakes low-volume queries.

Comparison Table

Cross-Encoder Rerankers

The workhorse of production reranking. Models encode [query, document] pairs and output a relevance score.

Popular models (2026):

For most RAG pipelines, ms-marco-MiniLM-L-6-v2 is the starting point — fast enough for real-time, good enough for 80% of use cases. Upgrade to bge-reranker-v2-m3 when eval shows MiniLM leaving quality on the table.

ColBERT as Reranker

ColBERT precomputes document token vectors and scores via late interaction at query time. Faster than cross-encoders on large candidate sets because document encoding happens at index time.

Use ColBERT reranking when:

• Candidate pool is 50-200 documents
• Cross-encoder latency exceeds budget at that pool size
• Corpus contains keyword-heavy content where token matching matters

LLM Reranking

Use the LLM itself to score relevance:

RERANK_PROMPT = """Rate the relevance of this document to the query.
Score 0-10. Return only the number.

Query: {query}
Document: {document}"""

LLM reranking produces the highest quality scores but at 10-50x the latency and cost of cross-encoders. Reserve for:

• Legal/compliance where precision is critical
• Low query volume (< 100/day)
• Queries where cross-encoder models lack domain vocabulary

For high-volume production, cross-encoders win on cost and latency. See LLM routing strategies for cost-aware model selection.

Building a Two-Stage RAG Pipeline in Python

Short answer: A production two-stage pipeline retrieves top-50 with a bi-encoder, reranks to top-5 with a cross-encoder, then sends those 5 chunks to the LLM.

Complete Pipeline

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np
from dataclasses import dataclass

@dataclass
class RetrievalConfig:
    bi_encoder_model: str = "BAAI/bge-large-en-v1.5"
    cross_encoder_model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"
    first_stage_k: int = 50
    final_k: int = 5

class TwoStageRetriever:
    def __init__(self, config: RetrievalConfig):
        self.config = config
        self.bi_encoder = SentenceTransformer(config.bi_encoder_model)
        self.cross_encoder = CrossEncoder(config.cross_encoder_model)
        self.documents: list[str] = []
        self.doc_embeddings: np.ndarray | None = None

    def index(self, documents: list[str]):
        self.documents = documents
        self.doc_embeddings = self.bi_encoder.encode(
            documents, normalize_embeddings=True, show_progress_bar=True,
        )

    def retrieve(self, query: str) -> list[dict]:
        query_embedding = self.bi_encoder.encode(
            [query], normalize_embeddings=True,
        )[0]
        similarities = np.dot(self.doc_embeddings, query_embedding)
        top_indices = np.argsort(similarities)[::-1][:self.config.first_stage_k]

        candidates = [
            {"doc_id": int(idx), "content": self.documents[idx], "bi_score": float(similarities[idx])}
            for idx in top_indices
        ]

        # Stage 2: Cross-encoder reranking
        pairs = [(query, c["content"]) for c in candidates]
        rerank_scores = self.cross_encoder.predict(pairs)

        for candidate, score in zip(candidates, rerank_scores):
            candidate["rerank_score"] = float(score)

        candidates.sort(key=lambda x: x["rerank_score"], reverse=True)
        return candidates[:self.config.final_k]

End-to-End RAG with Reranking

from openai import OpenAI

client = OpenAI()

def rag_with_reranking(query: str, retriever: TwoStageRetriever) -> dict:
    retrieved = retriever.retrieve(query)

    context = "\n\n---\n\n".join(
        f"[Source {r['doc_id']}]\n{r['content']}" for r in retrieved
    )

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question using ONLY the provided context. "
                    "Cite source IDs. If the context does not contain the answer, say so."
                ),
            },
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
        ],
    )

    return {
        "answer": response.choices[0].message.content,
        "sources": [{"doc_id": r["doc_id"], "score": r["rerank_score"]} for r in retrieved],
        "num_candidates": retriever.config.first_stage_k,
        "num_final": len(retrieved),
    }

Reranking with Cohere API

For teams preferring managed reranking over self-hosted cross-encoders:

import cohere

co = cohere.Client()

def cohere_rerank(query: str, documents: list[str], top_n: int = 5) -> list[dict]:
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=documents,
        top_n=top_n,
    )
    return [
        {
            "doc_id": r.index,
            "content": documents[r.index],
            "rerank_score": r.relevance_score,
        }
        for r in response.results
    ]

Cohere Rerank v3.5 supports 100+ languages and requires no GPU infrastructure — good fit for teams without ML ops capacity on cloud infrastructure.

Score Threshold Filtering

Not all reranked results deserve LLM context. Filter by score:

def retrieve_with_threshold(
    query: str,
    retriever: TwoStageRetriever,
    min_score: float = 0.3,
) -> list[dict]:
    results = retriever.retrieve(query)
    filtered = [r for r in results if r["rerank_score"] >= min_score]

    if not filtered:
        return []  # trigger "I don't know" response

    return filtered

Sending low-confidence chunks to the LLM causes hallucination. A score threshold of 0.3 (MiniLM) or 0.5 (bge-reranker) prevents this. Calibrate thresholds on your eval set.

Reranker Model Selection and Benchmarks

Short answer: Start with ms-marco-MiniLM-L-6-v2 for speed, upgrade to bge-reranker-v2-m3 if eval shows > 5% precision gap — always benchmark on your domain, not MS MARCO.

Domain Benchmark Results

We reranked 100 domain queries (same eval methodology as our RAG evaluation guide) across three corpora:

Reranking improved precision@5 by 15-28% across all corpora. The jump from no reranker to MiniLM-L-6-v2 is the largest gain — subsequent model upgrades yield diminishing returns.

Selection Decision Tree

Is recall@50 ≥ 0.85?
├── No → Fix retrieval first (chunking, embeddings, hybrid search)
└── Yes → Is latency budget > 500ms?
    ├── No → MiniLM-L-6-v2 (68ms rerank)
    └── Yes → Is multilingual?
        ├── Yes → bge-reranker-v2-m3 or Cohere rerank-v3.5
        └── No → MiniLM-L-12-v2 or bge-reranker-v2-m3

Evaluating Reranker Impact

def eval_reranker_impact(
    eval_queries: list[dict],
    retriever: TwoStageRetriever,
) -> dict:
    from rag_eval import precision_at_k  # from eval guide

    bi_encoder_only = []
    with_reranker = []

    for item in eval_queries:
        query = item["query"]
        relevant_ids = set(item["relevant_doc_ids"])

        # Without reranker: bi-encoder top-5
        query_emb = retriever.bi_encoder.encode([query], normalize_embeddings=True)[0]
        sims = np.dot(retriever.doc_embeddings, query_emb)
        bi_top5 = [str(i) for i in np.argsort(sims)[::-1][:5]]
        bi_encoder_only.append(precision_at_k(bi_top5, relevant_ids, 5))

        # With reranker: two-stage top-5
        reranked = retriever.retrieve(query)
        rerank_top5 = [str(r["doc_id"]) for r in reranked]
        with_reranker.append(precision_at_k(rerank_top5, relevant_ids, 5))

    return {
        "precision_at_5_without_reranker": sum(bi_encoder_only) / len(bi_encoder_only),
        "precision_at_5_with_reranker": sum(with_reranker) / len(with_reranker),
        "improvement": (sum(with_reranker) - sum(bi_encoder_only)) / len(bi_encoder_only),
    }

Run this before and after any reranker change. Store results in your eval artifact history.

Latency, Cost, and Scaling Strategies

Short answer: Cross-encoder reranking adds 50-150ms for 20 candidates on CPU — scale with batch inference, GPU acceleration, or async prefetching when query volume exceeds 20 QPS.

Latency Budget Breakdown

Reranking is 10-15% of total latency — not the bottleneck. But at 100+ QPS, reranker inference needs dedicated infrastructure.

GPU Acceleration

import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", device=device)

def batch_rerank(query: str, documents: list[str], batch_size: int = 32) -> list[float]:
    pairs = [(query, doc) for doc in documents]
    scores = cross_encoder.predict(pairs, batch_size=batch_size, show_progress_bar=False)
    return scores.tolist()

GPU batch inference reduces reranking 50 docs from ~200ms (CPU) to ~40ms (T4 GPU). Deploy reranker as a separate microservice on your backend API layer.

Caching Strategies

Query cache — Cache reranked results for identical queries (5-minute TTL). Hit rates of 15-30% on support bots.

Document score cache — For fixed query templates ("summarize policy X"), precompute rerank scores.

Approximate reranking — Use MiniLM for real-time queries, bge-reranker for cached/background reprocessing.

Cost Comparison

For > 10K queries/day, self-hosted cross-encoders are 5-10x cheaper than API reranking. For < 1K queries/day, Cohere API avoids infrastructure entirely.

Production Patterns and Anti-Patterns

Short answer: The production pattern is retrieve 50, rerank to 5, filter by score threshold, deduplicate, then generate — skip reranking only when eval proves it unnecessary.

Recommended Pipeline

Query → Embed → ANN Search (top-50)
                    ↓
              Cross-Encoder Rerank
                    ↓
              Score Threshold Filter (≥ 0.3)
                    ↓
              Deduplicate (remove >70% overlap)
                    ↓
              Top-5 Final Chunks → LLM Generation

Pattern 1: Cascaded Reranking

For high-stakes applications, cascade two rerankers:

def cascaded_rerank(query: str, candidates: list[str], fast_k: int = 20, final_k: int = 5):
    fast_reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
    precise_reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

    # Stage 1: Fast reranker narrows 50 → 20
    pairs = [(query, doc) for doc in candidates]
    fast_scores = fast_reranker.predict(pairs)
    ranked = sorted(zip(candidates, fast_scores), key=lambda x: x[1], reverse=True)
    top_20 = [doc for doc, _ in ranked[:fast_k]]

    # Stage 2: Precise reranker narrows 20 → 5
    precise_pairs = [(query, doc) for doc in top_20]
    precise_scores = precise_reranker.predict(precise_pairs)
    final = sorted(zip(top_20, precise_scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in final[:final_k]]

Total latency: ~70ms (fast) + ~60ms (precise) = ~130ms. Better quality than either alone.

Pattern 2: Reranking with Metadata Boost

Combine reranker scores with metadata signals:

def rerank_with_metadata(
    query: str,
    candidates: list[dict],
    cross_encoder: CrossEncoder,
    metadata_weight: float = 0.2,
) -> list[dict]:
    pairs = [(query, c["content"]) for c in candidates]
    rerank_scores = cross_encoder.predict(pairs)

    for candidate, score in zip(candidates, rerank_scores):
        metadata_boost = 0.0
        if candidate.get("doc_type") == "official_policy":
            metadata_boost += 0.1
        if candidate.get("last_updated_days", 999) < 30:
            metadata_boost += 0.05

        candidate["final_score"] = (1 - metadata_weight) * score + metadata_weight * metadata_boost

    candidates.sort(key=lambda x: x["final_score"], reverse=True)
    return candidates

Pair with self-querying retrieval when metadata filters pre-select candidates before reranking.

Anti-Pattern 1: Reranking Too Few Candidates

Reranking top-5 from bi-encoder misses documents ranked 6-50 that the reranker would promote. Always rerank at least 20-50 candidates.

Anti-Pattern 2: Reranking Before Fixing Recall

If recall@50 is 0.40, reranking 50 irrelevant documents produces 5 slightly-less-irrelevant documents. Fix retrieval first.

Anti-Pattern 3: Ignoring Score Calibration

Different reranker models produce different score distributions. A threshold of 0.3 for MiniLM is not equivalent to 0.3 for bge-reranker. Calibrate on your eval set.

Anti-Pattern 4: Reranking Duplicates

Bi-encoder retrieval often returns near-duplicate chunks. Reranking them wastes context window:

def deduplicate_chunks(chunks: list[dict], overlap_threshold: float = 0.7) -> list[dict]:
    from difflib import SequenceMatcher

    unique = []
    for chunk in chunks:
        is_duplicate = False
        for existing in unique:
            ratio = SequenceMatcher(None, chunk["content"], existing["content"]).ratio()
            if ratio > overlap_threshold:
                is_duplicate = True
                break
        if not is_duplicate:
            unique.append(chunk)
    return unique

Deduplicate after reranking, before LLM generation. This is step 5 in our RAG garbage debugging workflow.

Anti-Pattern 5: No Reranker Monitoring

Log reranker scores in production. Alert when average top-1 rerank score drops below threshold — indicates corpus drift or embedding model degradation. Wire into observability alongside retrieval metrics. For agent-based systems that retrieve across multiple tool calls, apply the same reranking pattern per retrieval step — see agentic workflows and agent memory patterns for multi-step retrieval architectures.

Primary references: official documentation, official documentation, official documentation, official documentation.

Frequently Asked Questions

What is reranking in RAG?

Reranking is a second-stage retrieval step that rescores first-stage candidates using a more accurate model (typically a cross-encoder) to improve the relevance ranking of chunks sent to the LLM.

How much does reranking improve RAG quality?

Reranking typically improves precision@5 by 15-25% when first-stage recall@50 is above 0.85. It does not improve recall — if the right document is not in the candidate pool, reranking cannot surface it.

Which reranker model should I use?

Start with cross-encoder/ms-marco-MiniLM-L-6-v2 for speed (68ms on 20 docs). Upgrade to BAAI/bge-reranker-v2-m3 or Cohere rerank-v3.5 if eval shows > 5% precision gap. Always benchmark on your domain data.

Does reranking add too much latency?

Cross-encoder reranking adds 50-150ms for 20 candidates on CPU, 30-60ms on GPU. This is 10-15% of total RAG latency. For sub-500ms requirements, use MiniLM-L-6-v2 or precompute scores for common queries.

Should I rerank before or after metadata filtering?

Filter first, rerank second. Self-querying retrieval and metadata filters reduce the candidate pool. Rerank the filtered set for best precision. Exception: when metadata filters are too aggressive and exclude relevant docs — then rerank before filtering.

Can I use an LLM as a reranker?

Yes, but LLM reranking costs 10-50x more than cross-encoders and adds 500ms-2s latency. Use LLM reranking only for low-volume, high-stakes queries. For production volume, cross-encoders or ColBERT are more cost-effective.

How many candidates should I rerank?

20-50 candidates is the production sweet spot. Fewer than 15 misses promotion opportunities. More than 100 hits latency limits with cross-encoders — use ColBERT for larger pools.

Do I need reranking if I use hybrid search?

Yes, in most cases. Hybrid search (BM25 + dense) improves recall but does not solve ranking precision. Reranking after fusion produces better top-5 results than either method alone. Measure with retrieval eval.

Conclusion

Reranking in RAG pipelines is the highest-ROI retrieval upgrade for most production systems:

• Retrieve 50 candidates with bi-encoder, rerank to 5 with cross-encoder
• Expect 15-25% precision improvement when recall@50 is already healthy
• Start with MiniLM-L-6-v2, upgrade only when eval proves the gap
• Filter by score threshold, deduplicate, then generate
• Monitor reranker scores in production for corpus drift

At HinterBuild, we add reranking to every RAG & LLM systems pipeline after retrieval eval confirms recall is healthy:

• RAG & LLM Systems
• Backend API Engineering
• Cloud Infrastructure & DevOps
• AI Agent Development

Schedule a consultation to optimize your retrieval pipeline.