Why Your RAG Pipeline Returns Garbage (And How to Fix It)

Table of Contents:

• Why RAG Pipelines Return Garbage
• Symptom 1: Chunking Failures
• Symptom 2: Embedding Mismatch
• Symptom 3: Retrieval Issues
• Symptom 4: Context Window Waste
• Symptom 5: Missing Reranking
• The Production RAG Debugging Workflow
• Before and After Metrics
• Frequently Asked Questions

Why RAG Pipelines Return Garbage

Short answer: When a RAG pipeline returns garbage, the LLM is usually innocent — broken retrieval upstream sends irrelevant, truncated, or duplicated chunks that force the model to hallucinate or say "I don't know."

I have debugged more broken RAG pipelines than I care to count at HinterBuild. The pattern is always the same: the team blames the LLM, swaps GPT-4 for Claude, tries fine-tuning, and gets the same garbage answers. The problem was never the model. It was chunking that split tables in half, an embedding model trained on English applied to multilingual docs, and retrieval returning 5 chunks that all said the same thing.

This guide covers the five root causes of bad RAG results and the production fixes we apply on every RAG & LLM systems engagement.

Key Takeaways:

• 80% of bad RAG results trace to chunking and embedding problems, not the LLM
• Fixed-size chunking destroys semantic boundaries — use structure-aware splitting
• Query and document embeddings must use the same model and version
• Retrieval without reranking returns plausible-but-wrong chunks 30-40% of the time
• Measure retrieval precision before tuning prompts or swapping models

Symptom 1: Chunking Failures

Short answer: Bad chunking splits documents at arbitrary character boundaries, separating questions from answers, headers from content, and code blocks from their explanations.

The Fixed-Size Chunking Trap

The most common cause of RAG pipeline garbage is naive fixed-size chunking:

def bad_chunk(text: str, chunk_size: int = 512) -> list[str]:
    return [text[i:i + chunk_size] for i in range(0, len(text), chunk_size)]

This splits a FAQ entry like:

Q: What is the refund policy for annual subscriptions?
A: Annual subscriptions are eligible for a prorated refund within 30 days...

Into:

Chunk 1: "Q: What is the refund policy for annual sub"
Chunk 2: "scriptions? A: Annual subscriptions are elig"
Chunk 3: "ible for a prorated refund within 30 days..."

Neither chunk is semantically complete. Retrieval returns chunk 2 for "refund policy" queries — the model sees a fragment and hallucinates the rest.

Structure-Aware Chunking Fix

from langchain_text_splitters import RecursiveCharacterTextSplitter
from dataclasses import dataclass

@dataclass
class ChunkConfig:
    chunk_size: int = 1000
    chunk_overlap: int = 200
    separators: list[str] = None

    def __post_init__(self):
        if self.separators is None:
            self.separators = ["\n## ", "\n### ", "\n\n", "\n", ". ", " "]

def structure_aware_chunk(text: str, config: ChunkConfig) -> list[dict]:
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=config.chunk_size,
        chunk_overlap=config.chunk_overlap,
        separators=config.separators,
        length_function=len,
    )

    raw_chunks = splitter.split_text(text)
    return [
        {
            "text": chunk,
            "metadata": {
                "chunk_index": i,
                "char_count": len(chunk),
                "has_complete_sentences": chunk.rstrip().endswith((".", "?", "!")),
            }
        }
        for i, chunk in enumerate(raw_chunks)
    ]

Chunking Strategy by Document Type

For PDF ingestion, never rely on raw text extraction alone — use layout-aware parsers (Unstructured, Docling) that preserve table and header structure.

Our backend API engineering team builds ingestion pipelines that handle each document type with the right splitter.

Symptom 2: Embedding Mismatch

Short answer: Embedding mismatch occurs when query and document embeddings use different models, versions, or preprocessing — making semantic similarity search return irrelevant results.

Common Embedding Mismatch Patterns

Pattern 1: Different models for indexing vs querying

# ❌ Indexed with text-embedding-ada-002, querying with text-embedding-3-small
index_embeddings = ada002.embed(documents)      # 1536 dimensions
query_embedding = embed_v3_small.embed(query)    # 1536 dimensions but different space!

Even with the same dimension count, different embedding models map text to different vector spaces. Similarity scores become meaningless.

Pattern 2: Language mismatch

Embedding multilingual-e5-large on English-only docs, then querying with German user questions. Cross-lingual retrieval requires models explicitly trained for it.

Pattern 3: Domain mismatch

General-purpose embeddings (text-embedding-3-small) applied to specialized domains (medical, legal, code). Domain-specific embeddings (voyage-code-2, bge-m3) outperform general models by 15-30% on specialized retrieval.

Embedding Consistency Fix

from enum import Enum

class EmbeddingModel(str, Enum):
    GENERAL = "text-embedding-3-small"
    CODE = "voyage-code-2"
    MULTILINGUAL = "multilingual-e5-large"

class EmbeddingPipeline:
    def __init__(self, model: EmbeddingModel):
        self.model = model
        self.client = self._init_client(model)

    async def embed_documents(self, texts: list[str]) -> list[list[float]]:
        """Embed documents for indexing — always use same model."""
        return await self.client.embed(texts, model=self.model.value)

    async def embed_query(self, query: str) -> list[float]:
        """Embed query — MUST use same model as indexing."""
        return (await self.client.embed([query], model=self.model.value))[0]

    async def reindex_required(self, new_model: EmbeddingModel) -> bool:
        """Changing embedding model requires full reindex."""
        return new_model != self.model

Critical rule: When you change embedding models, you must reindex the entire vector store. There is no migration path between embedding spaces.

Track embedding model version in chunk metadata for audit trails. Deploy embedding pipelines on cloud infrastructure with version pinning to prevent accidental model swaps.

Symptom 3: Retrieval Issues

Short answer: Retrieval issues — wrong top-k, missing metadata filters, no hybrid search — cause the RAG pipeline to fetch chunks that are semantically similar but factually irrelevant.

Problem: Pure Vector Search Misses Exact Matches

Vector search finds semantically similar text. It misses exact keyword matches that users expect:

User query: "error code E-4521"
Vector search returns: chunks about "error handling best practices"
Keyword search returns: chunk containing "E-4521: Database connection timeout"

The user needs the second result. Pure vector search never finds it.

Hybrid Search Fix

async def hybrid_retrieve(
    query: str,
    top_k: int = 10,
    vector_weight: float = 0.7,
    keyword_weight: float = 0.3,
    filters: dict = None,
) -> list[dict]:
    # Vector search
    query_embedding = await embed_pipeline.embed_query(query)
    vector_results = await vector_store.similarity_search(
        embedding=query_embedding,
        top_k=top_k * 2,  # Over-fetch for reranking
        filter=filters,
    )

    # Keyword search (BM25)
    keyword_results = await keyword_index.search(
        query=query,
        top_k=top_k * 2,
        filter=filters,
    )

    # Reciprocal Rank Fusion
    fused = reciprocal_rank_fusion(
        [vector_results, keyword_results],
        weights=[vector_weight, keyword_weight],
    )

    return fused[:top_k]


def reciprocal_rank_fusion(result_lists: list[list], weights: list[float], k: int = 60) -> list:
    """Combine ranked lists using RRF scoring."""
    scores = {}
    for results, weight in zip(result_lists, weights):
        for rank, item in enumerate(results):
            doc_id = item["id"]
            scores[doc_id] = scores.get(doc_id, 0) + weight / (k + rank + 1)

    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return [get_document_by_id(doc_id) for doc_id, _ in ranked]

Metadata Filtering Is Non-Negotiable

Without metadata filters, a multi-tenant RAG system returns chunks from every tenant:

# ❌ Returns chunks from all customers
results = await vector_store.search(query_embedding, top_k=5)

# ✅ Scoped to current tenant
results = await vector_store.search(
    query_embedding,
    top_k=5,
    filter={
        "tenant_id": current_tenant_id,
        "document_type": {"in": ["policy", "faq"]},
        "published_after": "2025-01-01",
    },
)

Always filter by tenant, document type, date range, and access level before similarity scoring. This prevents the most embarrassing bad RAG results — returning a competitor's data to the wrong customer.

For retrieval debugging, implement observability and monitoring that logs every query, retrieved chunks, and relevance scores.

Symptom 4: Context Window Waste

Short answer: Context window waste happens when retrieved chunks contain redundant information, boilerplate, or metadata that consumes tokens without adding answer value.

The Duplication Problem

Retrieving top-5 chunks from a 50-page document about the same topic returns five near-identical paragraphs. The LLM sees 2,500 tokens of repetition and 200 tokens of useful information.

def deduplicate_chunks(chunks: list[dict], similarity_threshold: float = 0.85) -> list[dict]:
    """Remove near-duplicate chunks before sending to LLM."""
    unique = []
    seen_embeddings = []

    for chunk in chunks:
        chunk_embedding = chunk["embedding"]
        is_duplicate = any(
            cosine_similarity(chunk_embedding, seen) > similarity_threshold
            for seen in seen_embeddings
        )
        if not is_duplicate:
            unique.append(chunk)
            seen_embeddings.append(chunk_embedding)

    return unique

Context Compression

For large retrieved sets, compress context before sending to the LLM:

async def compress_context(query: str, chunks: list[dict], max_tokens: int = 3000) -> str:
    """Extract only query-relevant sentences from retrieved chunks."""
    compressed_parts = []
    token_count = 0

    for chunk in chunks:
        sentences = chunk["text"].split(". ")
        for sentence in sentences:
            relevance = await score_sentence_relevance(query, sentence)
            if relevance > 0.5 and token_count + len(sentence) < max_tokens:
                compressed_parts.append(sentence)
                token_count += len(sentence)

    return ". ".join(compressed_parts)

This reduces token costs 40-60% while improving answer quality — the LLM sees dense, relevant context instead of padded chunks.

Understand when RAG is the right approach vs fine-tuning in our RAG vs fine-tuning guide.

Symptom 5: Missing Reranking

Short answer: Without a reranking step, vector search returns chunks that are topically related but not answer-relevant — the single biggest cause of plausible-but-wrong RAG answers.

Why Vector Search Alone Fails

Vector search optimizes for semantic similarity — "how close is this chunk's meaning to the query?" Reranking optimizes for relevance — "does this chunk actually answer the query?"

In our benchmarks across 12 production RAG systems:

Cross-Encoder Reranking Fix

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

async def retrieve_and_rerank(
    query: str,
    top_k_retrieve: int = 20,
    top_k_final: int = 5,
    filters: dict = None,
) -> list[dict]:
    # Stage 1: Over-fetch with hybrid search
    candidates = await hybrid_retrieve(
        query=query,
        top_k=top_k_retrieve,
        filters=filters,
    )

    # Stage 2: Rerank with cross-encoder
    pairs = [(query, chunk["text"]) for chunk in candidates]
    scores = reranker.predict(pairs)

    ranked = sorted(
        zip(candidates, scores),
        key=lambda x: x[1],
        reverse=True,
    )

    # Stage 3: Deduplicate and return top-k
    final = deduplicate_chunks([chunk for chunk, _ in ranked])
    return final[:top_k_final]

Reranking adds 50-150ms latency but improves answer quality more than swapping from GPT-4o-mini to GPT-4o. Always rerank before blaming the LLM.

For production deployments, use dedicated reranking APIs (Cohere Rerank, Jina Reranker) that handle batching and GPU inference. Self-hosted cross-encoders work for lower query volumes.

The Production RAG Debugging Workflow

Short answer: Debug RAG pipelines bottom-up — fix retrieval before touching prompts, fix chunking before fixing retrieval.

Step 1: Evaluate Retrieval in Isolation

Before changing anything about the LLM, measure retrieval quality:

async def evaluate_retrieval(test_cases: list[dict]) -> dict:
    """Test retrieval without LLM generation."""
    results = {"precision_at_5": [], "mrr": [], "failures": []}

    for case in test_cases:
        # case = {"query": str, "expected_doc_ids": list[str]}
        retrieved = await retrieve_and_rerank(case["query"], top_k_final=5)
        retrieved_ids = [c["metadata"]["doc_id"] for c in retrieved]

        hits = len(set(retrieved_ids) & set(case["expected_doc_ids"]))
        precision = hits / min(len(retrieved_ids), 5)
        results["precision_at_5"].append(precision)

        # Mean Reciprocal Rank
        for rank, doc_id in enumerate(retrieved_ids):
            if doc_id in case["expected_doc_ids"]:
                results["mrr"].append(1 / (rank + 1))
                break
        else:
            results["mrr"].append(0)
            results["failures"].append(case)

    return {
        "avg_precision_at_5": sum(results["precision_at_5"]) / len(results),
        "avg_mrr": sum(results["mrr"]) / len(results),
        "failure_cases": results["failures"],
    }

Target metrics:

• Precision@5 > 0.75 before adding reranking
• Precision@5 > 0.85 after reranking
• MRR > 0.70

If retrieval precision is below 0.60, the problem is chunking or embeddings — not retrieval parameters.

Step 2: Inspect Failure Cases

For each failed retrieval, check:

1. Is the correct chunk in the index? (Ingestion failure)
2. Is the chunk semantically complete? (Chunking failure)
3. Does the query embedding match the chunk embedding space? (Embedding mismatch)
4. Is the correct chunk in top-20 but not top-5? (Reranking needed)
5. Is metadata filtering excluding the right chunk? (Filter bug)

Step 3: Fix Bottom-Up

Fix order (never skip steps):
1. Chunking → reindex
2. Embedding model → reindex
3. Hybrid search → no reindex needed
4. Reranking → no reindex needed
5. Prompt tuning → last resort
6. Model swap → absolute last resort

Teams that skip to step 5 or 6 waste weeks. The RAG vs fine-tuning decision matters here — if retrieval is broken, fine-tuning makes it worse.

Integrate retrieval metrics into your AI agent observability stack alongside generation quality scores.

Before and After Metrics

Short answer: A properly tuned RAG pipeline improves retrieval precision from ~55% to ~90% and answer faithfulness from ~60% to ~85% — without changing the LLM.

Case Study: Legal Document Q&A System

A client came to HinterBuild with a legal document Q&A system returning bad RAG results on 45% of queries. Their diagnosis: "GPT-4 is not smart enough for legal text."

Before fixes:

Root causes found:

1. Fixed 512-token chunking split legal clauses mid-sentence
2. Indexed with text-embedding-ada-002, queried with text-embedding-3-small
3. No metadata filtering (returns docs from wrong jurisdictions)
4. No reranking step
5. Top-5 chunks with 70% content overlap

After fixes:

Same LLM. Same documents. Fixed pipeline.

Build production-grade RAG with our RAG & LLM systems team. Pair with hallucination reduction techniques for grounded, cited answers.

For agent-based RAG systems, see agentic workflows and agent memory patterns.

Contact us to audit your RAG pipeline.

Primary references: official documentation, official documentation, official documentation, official documentation.

Frequently Asked Questions

Why does my RAG system give wrong answers?

Wrong answers in RAG systems almost always trace to retrieval failure — the LLM generates plausible text from irrelevant context. Fix chunking, embedding consistency, and add reranking before changing the LLM model.

How do I know if my chunks are too small or too large?

Test with your evaluation set. Chunks are too small if retrieval returns fragments that lack context (precision is OK but answers are incomplete). Chunks are too large if retrieval returns chunks with low relevance scores and the LLM ignores most of the content. Target 500-1000 tokens for most document types.

Should I use the same embedding model for indexing and querying?

Always yes. Using different embedding models — even different versions of the same model — produces incompatible vector spaces. Similarity scores become meaningless. If you change embedding models, reindex everything.

What is the best chunk size for RAG?

There is no universal best size. Use structure-aware chunking matched to document type: 200-500 tokens for FAQ, 500-1000 for technical docs, whole-table extraction for tabular data. Overlap of 10-20% prevents boundary information loss.

How many chunks should I retrieve?

Retrieve 15-25 candidates, rerank to 3-5 final chunks. Retrieving fewer than 3 limits answer completeness. Retrieving more than 7 without deduplication wastes context window and introduces noise.

Does reranking add too much latency?

Cross-encoder reranking adds 50-150ms for 20 candidates — acceptable for most applications. The quality improvement (15-25% precision gain) far outweighs the latency cost. For sub-500ms requirements, use lightweight rerankers like ms-marco-MiniLM-L-6-v2.

Can I fix bad RAG results by fine-tuning the LLM?

No. Fine-tuning teaches output patterns, not retrieval quality. If the wrong chunks reach the LLM, a fine-tuned model will confidently generate wrong answers in your preferred format. Fix retrieval first, then consider fine-tuning for output consistency.

How do I evaluate RAG pipeline quality?

Build a test set of 50-100 query-answer pairs with known source documents. Measure retrieval precision@k, mean reciprocal rank (MRR), answer faithfulness (does the answer match retrieved context?), and answer relevance (does it address the query?). Track these metrics continuously in production.

Conclusion

When your RAG pipeline returns garbage, resist the urge to swap LLMs or start fine-tuning. Debug bottom-up:

1. Chunking — structure-aware splitting matched to document type
2. Embeddings — same model for indexing and querying, domain-appropriate
3. Retrieval — hybrid search with metadata filtering
4. Reranking — cross-encoder reranking before LLM generation
5. Context — deduplicate and compress retrieved chunks
6. LLM — tune prompts only after retrieval precision exceeds 85%

The pipeline matters more than the model. We have seen GPT-4o-mini with a tuned pipeline outperform GPT-4o with a broken one.

At HinterBuild:

• RAG & LLM Systems
• AI Agent Development
• Backend API Engineering
• Observability & Monitoring

Schedule a RAG pipeline audit — we will find the bottleneck in your first session.