RAG Evaluation Without Ground Truth: Practical Guide
RAG evaluation without labeled data — LLM-as-judge, reference-free metrics, retrieval quality measurement, and production monitoring patterns.
Muhammad Abdul Sami
· 9 min read
- RAG
- Embeddings
- Vector Databases
- Evaluation
Table of Contents:
- Why Evaluation Without Ground Truth Matters
- LLM-as-Judge for Answer Quality
- Retrieval Quality Metrics
- Reference-Free Evaluation Patterns
- Production Monitoring
- Automated Quality Regression Detection
- Building an Evaluation Pipeline
- Cost and Latency Considerations
- Frequently Asked Questions
Why Evaluation Without Ground Truth Matters
Short answer: Most production RAG systems lack labeled ground-truth answers for every query. Reference-free evaluation uses LLM-as-judge, retrieval quality proxies, and self-consistency checks to measure RAG quality without human labels.
Building RAG at HinterBuild, clients ask: "How do we know if our RAG system is working?" They have 50K documents, 10K queries/month, and zero labeled test sets. Creating ground truth costs $50K+ and goes stale in months. Reference-free evaluation provides continuous quality measurement without manual labeling.
Key Takeaways:
- Ground truth creation costs $5-20 per labeled query-answer pair — unsustainable at scale
- LLM-as-judge evaluates answer quality at $0.001-0.003 per query (500x cheaper)
- Retrieval metrics (score distribution, chunk diversity, citation coverage) predict answer quality
- Self-consistency (answer k times, check agreement) detects hallucinations without labels
- Production monitoring tracks quality drift before user complaints
- Sample 1-5% of queries for human review to calibrate automated metrics
A fintech client had no test set for their 80K-document compliance RAG. We deployed LLM-as-judge + retrieval monitoring. When answer quality scores dropped 12% week-over-week, investigation found a chunking regression. Fixed before users noticed.
LLM-as-Judge for Answer Quality
LLM-as-judge uses a strong LLM (GPT-4o, Claude 3.5 Sonnet) to evaluate answer quality without ground truth.
Basic LLM-as-Judge Pattern
from openai import OpenAI
client = OpenAI()
async def evaluate_answer_quality(question: str, answer: str, context: str) -> dict:
"""Evaluate answer quality without ground truth."""
eval_prompt = f"""Evaluate the answer quality on these criteria:
1. **Relevance** (0-10): Does the answer address the question?
2. **Faithfulness** (0-10): Is the answer supported by the context?
3. **Completeness** (0-10): Does the answer fully address all parts of the question?
4. **Clarity** (0-10): Is the answer clear and well-structured?
Question: {question}
Context:
{context}
Answer:
{answer}
Return JSON:
{{
"relevance": 0-10,
"faithfulness": 0-10,
"completeness": 0-10,
"clarity": 0-10,
"overall_score": 0-10,
"reasoning": "Brief explanation",
"issues": ["List of issues if score < 7"]
}}
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are an expert evaluator of question-answering systems."},
{"role": "user", "content": eval_prompt}
],
response_format={"type": "json_object"},
temperature=0.1
)
return json.loads(response.choices[0].message.content)
question = "What is the refund policy?"
answer = "Refunds are available within 30 days of purchase with receipt."
context = "...[retrieved chunks]..."
eval_result = await evaluate_answer_quality(question, answer, context)
print(f"Overall score: {eval_result['overall_score']}/10")
print(f"Issues: {eval_result['issues']}")
Advanced: Multi-Aspect Evaluation
async def multi_aspect_evaluation(question: str, answer: str, context: str) -> dict:
"""Evaluate multiple aspects with separate LLM calls for reliability."""
# Aspect 1: Faithfulness (grounding in context)
faithfulness_score = await evaluate_faithfulness(answer, context)
# Aspect 2: Answer Relevance
relevance_score = await evaluate_relevance(question, answer)
# Aspect 3: Context Relevance (was retrieval good?)
context_relevance = await evaluate_context_relevance(question, context)
# Aspect 4: Hallucination Detection
hallucination_check = await detect_hallucinations(answer, context)
return {
"faithfulness": faithfulness_score,
"answer_relevance": relevance_score,
"context_relevance": context_relevance,
"hallucination_detected": hallucination_check["hallucinated"],
"overall_score": (faithfulness_score + relevance_score + context_relevance) / 3,
}
async def evaluate_faithfulness(answer: str, context: str) -> float:
"""Check if answer is supported by context."""
prompt = f"""Does the following answer contain ONLY information present in the context?
Context:
{context}
Answer:
{answer}
Return JSON: {{"faithfulness_score": 0-10, "unsupported_claims": [list]}}
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0.0
)
result = json.loads(response.choices[0].message.content)
return result["faithfulness_score"]
async def evaluate_relevance(question: str, answer: str) -> float:
"""Check if answer addresses the question."""
prompt = f"""Rate how well the answer addresses the question (0-10).
Question: {question}
Answer: {answer}
Return JSON: {{"relevance_score": 0-10, "reasoning": str}}
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0.0
)
result = json.loads(response.choices[0].message.content)
return result["relevance_score"]
async def evaluate_context_relevance(question: str, context: str) -> float:
"""Check if retrieved context is relevant to question."""
prompt = f"""Rate how relevant the context is to answering the question (0-10).
Question: {question}
Context: {context}
Return JSON: {{"context_relevance": 0-10, "relevant_chunks": int, "total_chunks": int}}
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0.0
)
result = json.loads(response.choices[0].message.content)
return result["context_relevance"]
LLM-as-Judge Calibration
LLM-as-judge correlates 0.75-0.85 with human judgment when calibrated. Validate on 50-100 human-labeled examples first.
async def calibrate_llm_judge(test_set: list[dict]) -> dict:
"""Calibrate LLM-as-judge against human labels."""
llm_scores = []
human_scores = []
for example in test_set:
llm_eval = await evaluate_answer_quality(
example["question"],
example["answer"],
example["context"]
)
llm_scores.append(llm_eval["overall_score"])
human_scores.append(example["human_score"])
# Calculate correlation
correlation = np.corrcoef(llm_scores, human_scores)[0, 1]
return {
"correlation": correlation,
"mean_absolute_error": np.mean(np.abs(np.array(llm_scores) - np.array(human_scores))),
"bias": np.mean(np.array(llm_scores) - np.array(human_scores)),
}
Use with agentic RAG to evaluate iterative retrieval quality.
Retrieval Quality Metrics
Measure retrieval quality using only system-observable metrics (no labels needed).
Metric 1: Retrieval Score Distribution
def analyze_retrieval_scores(chunks: list[dict]) -> dict:
"""Analyze retrieval score distribution as quality proxy."""
scores = [c["score"] for c in chunks]
return {
"mean_score": np.mean(scores),
"min_score": np.min(scores),
"max_score": np.max(scores),
"score_variance": np.var(scores),
"score_gap": scores[0] - scores[-1] if len(scores) > 1 else 0, # Top vs last
}
# Good retrieval: high mean, low variance, large gap
# Bad retrieval: low mean, high variance, small gap (all equally mediocre)
Metric 2: Chunk Diversity
from sklearn.metrics.pairwise import cosine_similarity
def measure_chunk_diversity(chunk_embeddings: list[list[float]]) -> float:
"""High diversity = chunks cover different aspects of query."""
similarities = cosine_similarity(chunk_embeddings)
# Average similarity between chunks (excluding self-similarity)
n = len(similarities)
total_sim = similarities.sum() - n # Subtract diagonal
avg_similarity = total_sim / (n * (n - 1)) if n > 1 else 0
diversity = 1 - avg_similarity
return diversity
# High diversity (>0.6) = good coverage
# Low diversity (<0.3) = redundant chunks
Metric 3: Citation Coverage
def measure_citation_coverage(answer: str, chunks: list[dict]) -> dict:
"""Check if answer uses multiple retrieved sources."""
# Simple heuristic: count unique chunk references in answer
cited_chunks = set()
for i, chunk in enumerate(chunks):
# Check if chunk content appears in answer (simplified)
if any(phrase in answer.lower() for phrase in chunk["content"].lower().split()[:10]):
cited_chunks.add(i)
return {
"citation_rate": len(cited_chunks) / len(chunks) if chunks else 0,
"cited_chunks": len(cited_chunks),
"total_chunks": len(chunks),
}
# High citation rate (>0.6) = answer uses retrieved context
# Low rate (<0.3) = answer may be hallucinated or only uses one chunk
Metric 4: Context Sufficiency
async def check_context_sufficiency(question: str, chunks: list[dict]) -> dict:
"""LLM checks if context is sufficient to answer."""
context = "\n\n---\n\n".join(c["content"] for c in chunks)
prompt = f"""Can the question be fully answered using ONLY the provided context?
Question: {question}
Context: {context}
Return JSON: {{"sufficient": bool, "missing_info": [list], "confidence": 0-10}}
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0.0
)
return json.loads(response.choices[0].message.content)
Track retrieval metrics with observability and monitoring.
Reference-Free Evaluation Patterns
Pattern 1: Self-Consistency Check
Generate answer k times (k=3-5), check if answers agree. Disagreement indicates uncertainty/hallucination.
async def self_consistency_check(question: str, context: str, k: int = 3) -> dict:
"""Generate k answers, check consistency."""
answers = []
for i in range(k):
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Answer using only provided context."},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
],
temperature=0.7, # Some variation
)
answers.append(response.choices[0].message.content)
# Check agreement
agreement_scores = []
for i in range(k):
for j in range(i+1, k):
agreement = await measure_semantic_similarity(answers[i], answers[j])
agreement_scores.append(agreement)
avg_agreement = np.mean(agreement_scores)
return {
"self_consistency_score": avg_agreement,
"answers": answers,
"reliable": avg_agreement > 0.75, # High agreement = reliable
}
async def measure_semantic_similarity(text1: str, text2: str) -> float:
"""Measure semantic similarity between two texts."""
embeddings = client.embeddings.create(
input=[text1, text2],
model="text-embedding-3-small"
)
emb1 = embeddings.data[0].embedding
emb2 = embeddings.data[1].embedding
return np.dot(emb1, emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2))
Pattern 2: Claim Verification
Extract claims from answer, verify each against context.
async def verify_claims(answer: str, context: str) -> dict:
"""Extract claims and verify against context."""
# Step 1: Extract claims
extract_prompt = f"""Extract factual claims from the answer. Return JSON: {{"claims": [list]}}
Answer: {answer}
"""
claims_response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": extract_prompt}],
response_format={"type": "json_object"}
)
claims = json.loads(claims_response.choices[0].message.content)["claims"]
# Step 2: Verify each claim
verified_claims = []
for claim in claims:
verify_prompt = f"""Is this claim supported by the context?
Claim: {claim}
Context: {context}
Return JSON: {{"supported": bool, "confidence": 0-10}}
"""
verify_response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": verify_prompt}],
response_format={"type": "json_object"}
)
result = json.loads(verify_response.choices[0].message.content)
verified_claims.append({
"claim": claim,
"supported": result["supported"],
"confidence": result["confidence"]
})
# Calculate verification rate
supported_count = sum(1 for c in verified_claims if c["supported"])
verification_rate = supported_count / len(claims) if claims else 1.0
return {
"verification_rate": verification_rate,
"total_claims": len(claims),
"supported_claims": supported_count,
"unsupported_claims": [c["claim"] for c in verified_claims if not c["supported"]],
}
Pattern 3: Contradiction Detection
Check if answer contradicts retrieved context.
async def detect_contradictions(answer: str, context: str) -> dict:
"""Detect if answer contradicts context."""
prompt = f"""Does the answer contradict any information in the context?
Context:
{context}
Answer:
{answer}
Return JSON: {{"contradicts": bool, "contradictions": [list], "severity": "none"|"minor"|"major"}}
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0.0
)
return json.loads(response.choices[0].message.content)
Use with corrective RAG for self-correction loops.
Production Monitoring
Real-Time Quality Dashboard
from dataclasses import dataclass
from datetime import datetime, timedelta
@dataclass
class RAGQualityMetrics:
timestamp: datetime
avg_answer_score: float
avg_retrieval_score: float
avg_faithfulness: float
hallucination_rate: float
query_count: int
class RAGQualityMonitor:
def __init__(self, eval_sample_rate: float = 0.05): # Evaluate 5% of queries
self.sample_rate = eval_sample_rate
self.metrics_buffer = []
async def monitor_query(self, question: str, answer: str, chunks: list[dict], query_id: str):
"""Sample and evaluate queries in production."""
if random.random() > self.sample_rate:
return # Skip evaluation for this query
# Evaluate quality
context = "\n\n---\n\n".join(c["content"] for c in chunks)
eval_result = await evaluate_answer_quality(question, answer, context)
retrieval_analysis = analyze_retrieval_scores(chunks)
faithfulness = await evaluate_faithfulness(answer, context)
# Log metrics
metrics = {
"query_id": query_id,
"timestamp": datetime.utcnow(),
"answer_score": eval_result["overall_score"],
"retrieval_score": retrieval_analysis["mean_score"],
"faithfulness": faithfulness,
}
await self._log_metrics(metrics)
async def get_quality_report(self, hours: int = 24) -> dict:
"""Generate quality report for last N hours."""
cutoff = datetime.utcnow() - timedelta(hours=hours)
# Query metrics from last N hours
recent_metrics = await self._fetch_metrics_since(cutoff)
if not recent_metrics:
return {"error": "No metrics available"}
return {
"period_hours": hours,
"queries_evaluated": len(recent_metrics),
"avg_answer_score": np.mean([m["answer_score"] for m in recent_metrics]),
"avg_retrieval_score": np.mean([m["retrieval_score"] for m in recent_metrics]),
"avg_faithfulness": np.mean([m["faithfulness"] for m in recent_metrics]),
"low_quality_rate": sum(1 for m in recent_metrics if m["answer_score"] < 6) / len(recent_metrics),
}
async def _log_metrics(self, metrics: dict):
"""Log to time-series database (InfluxDB, Prometheus, etc.)"""
# Send to monitoring backend
pass
async def _fetch_metrics_since(self, cutoff: datetime) -> list[dict]:
"""Fetch metrics from monitoring backend."""
# Query from time-series DB
pass
Alert on Quality Degradation
class QualityAlertSystem:
def __init__(self, threshold_score: float = 7.0, threshold_drop: float = 0.15):
self.threshold_score = threshold_score
self.threshold_drop = threshold_drop
self.baseline_score = None
async def check_quality(self, current_metrics: dict):
"""Alert if quality drops below threshold."""
current_score = current_metrics["avg_answer_score"]
# Alert if below absolute threshold
if current_score < self.threshold_score:
await self._send_alert({
"severity": "high",
"message": f"RAG quality dropped to {current_score:.2f} (threshold: {self.threshold_score})",
"metrics": current_metrics,
})
# Alert if relative drop from baseline
if self.baseline_score and (self.baseline_score - current_score) / self.baseline_score > self.threshold_drop:
await self._send_alert({
"severity": "medium",
"message": f"RAG quality dropped {((self.baseline_score - current_score) / self.baseline_score * 100):.1f}% from baseline",
"metrics": current_metrics,
})
async def _send_alert(self, alert: dict):
"""Send alert via email, Slack, PagerDuty, etc."""
print(f"ALERT: {alert['message']}")
# Integrate with alerting system
Deploy with cloud infrastructure for reliable monitoring.
Automated Quality Regression Detection
Golden Query Set Testing
class GoldenSetEvaluator:
def __init__(self, golden_queries: list[dict]):
"""
golden_queries: [{"question": str, "expected_keywords": list, "expected_sources": list}, ...]
"""
self.golden_queries = golden_queries
async def run_regression_test(self, rag_system) -> dict:
"""Run golden query set, detect regressions."""
results = []
for query in self.golden_queries:
response = await rag_system.query(query["question"])
# Check if expected keywords present
keyword_match = sum(
1 for kw in query.get("expected_keywords", [])
if kw.lower() in response["answer"].lower()
) / len(query.get("expected_keywords", [])) if query.get("expected_keywords") else 1.0
# Check if expected sources retrieved
source_match = sum(
1 for src in query.get("expected_sources", [])
if src in [s["metadata"]["source"] for s in response.get("sources", [])]
) / len(query.get("expected_sources", [])) if query.get("expected_sources") else 1.0
results.append({
"question": query["question"],
"keyword_match": keyword_match,
"source_match": source_match,
"passed": keyword_match > 0.7 and source_match > 0.5,
})
pass_rate = sum(1 for r in results if r["passed"]) / len(results)
return {
"total_queries": len(results),
"pass_rate": pass_rate,
"failed_queries": [r for r in results if not r["passed"]],
}
Building an Evaluation Pipeline
Complete Evaluation Pipeline
class RAGEvaluationPipeline:
def __init__(self, rag_system, monitor, golden_set, alert_system):
self.rag_system = rag_system
self.monitor = monitor
self.golden_set = golden_set
self.alert_system = alert_system
async def evaluate_query(self, question: str, query_id: str) -> dict:
"""Full evaluation pipeline for one query."""
# Step 1: Execute RAG query
result = await self.rag_system.query(question)
# Step 2: Evaluate answer quality (LLM-as-judge)
context = "\n\n---\n\n".join(c["content"] for c in result["chunks"])
answer_eval = await evaluate_answer_quality(question, result["answer"], context)
# Step 3: Check retrieval quality
retrieval_metrics = analyze_retrieval_scores(result["chunks"])
# Step 4: Self-consistency check (for important queries)
consistency = await self_consistency_check(question, context, k=3)
# Step 5: Verify claims
verification = await verify_claims(result["answer"], context)
# Step 6: Log to monitoring
await self.monitor.monitor_query(question, result["answer"], result["chunks"], query_id)
return {
"answer": result["answer"],
"answer_quality": answer_eval["overall_score"],
"retrieval_quality": retrieval_metrics["mean_score"],
"self_consistency": consistency["self_consistency_score"],
"verification_rate": verification["verification_rate"],
"overall_confidence": self._calculate_confidence(answer_eval, retrieval_metrics, consistency, verification),
}
def _calculate_confidence(self, answer_eval, retrieval_metrics, consistency, verification) -> float:
"""Aggregate metrics into overall confidence score."""
return (
answer_eval["overall_score"] * 0.3 +
retrieval_metrics["mean_score"] * 10 * 0.2 +
consistency["self_consistency_score"] * 10 * 0.2 +
verification["verification_rate"] * 10 * 0.3
) / 10
async def run_daily_eval(self):
"""Run daily golden set evaluation + quality report."""
# Run golden set tests
regression_results = await self.golden_set.run_regression_test(self.rag_system)
# Get quality metrics
quality_report = await self.monitor.get_quality_report(hours=24)
# Check for alerts
await self.alert_system.check_quality(quality_report)
return {
"regression_tests": regression_results,
"quality_metrics": quality_report,
"timestamp": datetime.utcnow(),
}
Cost and Latency Considerations
Evaluation Cost Breakdown
| Method | Cost per Query | Latency Added |
|---|---|---|
| Retrieval metrics | $0 | ~5ms |
| LLM-as-judge (basic) | $0.001-0.003 | 300-800ms |
| Multi-aspect eval | $0.003-0.008 | 1-2s |
| Self-consistency (k=3) | $0.015-0.030 | 3-5s |
| Claim verification | $0.005-0.015 | 1-3s |
Cost-Effective Sampling Strategy
# Production monitoring budget: $500/month
# Average query cost: $0.005 (RAG) + $0.003 (eval) = $0.008
# Monthly queries: 100K
# Can afford to evaluate: $500 / $0.003 = 166K evaluations
# Strategy: Sample 5% of queries for full eval
# Sample 100% for retrieval metrics (free)
# Run golden set daily (100 queries * 30 days = 3K/month)
EVAL_BUDGET = {
"production_sampling": 0.05, # 5% of live traffic
"golden_set_frequency": "daily",
"golden_set_size": 100,
}
Related implementation guides:
Primary references: official documentation, official documentation, official documentation, official documentation.
Operating RAG Evaluation Without Ground Truth as a System
The implementation is only one part of RAG Evaluation Without Ground Truth. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.
Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.
Define the failure model
List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.
Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.
Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.
Measure the decision, not just the component
Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.
Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.
Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.
Roll out with reversible steps
Ship RAG Evaluation Without Ground Truth behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.
Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.
Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.
Capacity and cost controls
Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.
Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.
Production readiness review
Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.
The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our RAG Evaluation Without Ground Truth engineering support.
Frequently Asked Questions
How do I evaluate RAG quality without labeled data?
Use LLM-as-judge for answer quality, retrieval score analysis for search quality, and self-consistency checks for reliability. These reference-free methods work without ground-truth labels.
What is LLM-as-judge in RAG evaluation?
LLM-as-judge uses a strong LLM (GPT-4o, Claude) to evaluate answer quality against criteria like relevance, faithfulness, and completeness. Correlates 0.75-0.85 with human judgment at 500x lower cost.
How accurate is LLM-as-judge?
75-85% correlation with human judgment when calibrated. Validate on 50-100 human-labeled examples first. More reliable for binary judgments (good/bad) than fine-grained scoring (1-10).
What is self-consistency checking?
Self-consistency generates the same answer k times (k=3-5) with variation, then checks if answers agree. High agreement (>0.75) indicates reliable answer; low agreement (<0.5) indicates uncertainty or hallucination.
How much does RAG evaluation cost?
Retrieval metrics are free (system-observable). LLM-as-judge costs $0.001-0.003 per query. Full evaluation pipeline costs $0.005-0.015 per query. Sample 5-10% of production traffic to stay under budget.
How often should I evaluate production RAG?
Real-time: Track retrieval scores for 100% of queries (free).
Sampled: Evaluate 5-10% of queries with LLM-as-judge.
Daily: Run golden query set regression tests.
Weekly: Human review of 50-100 sampled queries.
What metrics predict RAG answer quality?
Retrieval score mean (>0.75 good), chunk diversity (>0.6 good), citation coverage (>0.6 good), faithfulness score (>8/10 good). These correlate 0.6-0.7 with answer quality.
Should I use human evaluation?
Yes, but sparingly. Use LLM-as-judge for scale, human eval for calibration and edge cases. Sample 1-5% of queries monthly for human review ($5-20 per query). Update calibration set quarterly.
Conclusion
RAG evaluation without ground truth relies on:
| Method | Use Case | Cost |
|---|---|---|
| LLM-as-judge | Answer quality at scale | $0.001-0.003/query |
| Retrieval metrics | Real-time quality proxy | Free |
| Self-consistency | Hallucination detection | $0.015-0.030/query |
| Golden set testing | Regression detection | $0.30-3.00/run |
| Human review | Calibration & edge cases | $5-20/query |
Start with retrieval metrics + LLM-as-judge on 5% of queries. Add golden set testing weekly. Validate with human review monthly.
At HinterBuild:
Schedule a consultation to design your RAG evaluation strategy.
Free consultation
Book a free consultation call on RAG evaluation & quality measurement
30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.
Book a meeting
Keep reading
Related articles
RAG Evaluation: How to Measure Retrieval Quality Before
RAG Evaluation guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.
Read post
RAGAS Deep Dive: Faithfulness & Relevancy Metrics for RAG
RAGAS Deep Dive guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.
Read post
RAG for Structured Data: Natural Language to SQL Guide
Learn rag for structured data through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post
RAG Pipeline Observability & Tracing
Learn rag pipeline observability & tracing through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post