RAG Evaluation: How to Measure Retrieval Quality (2026 Guide)

The Two Layers of RAG Quality

The debugging rule follows directly: if recall@k is low, no prompt will save you. If recall is high but faithfulness is low, the model is ignoring good context. Knowing which layer is broken halves your debugging time — and in privacy-first legal AI, where a hallucinated clause is a liability, this separation is non-negotiable.

Retrieval Metrics, Explained Without Math Anxiety

Retrieval metrics only need rankings + labeled relevant chunks — no LLM judge required. Write the labels once per question; the scores are then pure computation.

Generation Metrics: Faithfulness and Relevance

• Faithfulness — every claim in the answer is supported by the retrieved context. A 0.95-faithful system still hallucinates ~1 sentence in 20; for legal/medical work you want 0.99+ or a refusal policy.
• Answer relevance — the answer actually addresses the question. A faithful answer to the wrong question scores low here.
• Context precision/recall (RAGAS-style) — bridging metrics: how much of the retrieved context was actually needed, and how much of what was needed got retrieved.

These are typically scored by an LLM-as-judge. In 2026, a local sub-12B model judges faithfully on structured scales, so the whole evaluation loop stays on your machine — matching the privacy posture this blog covers in regulated-industry AI.

Building an Eval Set (Without Fake Data)

1. Mine real questions. User logs, support tickets, domain experts. 30–50 questions covering the range of real queries.
2. Label the gold chunks. For each question, mark which chunk(s) contain the answer. This is the expensive, honest part — and it's what makes the numbers trustworthy.
3. Synthetic augmentation (optional). RAGAS-style generation: sample chunks, ask an LLM to write questions only those chunks can answer. Good for coverage; always spot-check by hand, because synthetic questions are unrealistically aligned with the chunks.
4. Keep it versioned. Your eval set is a regression test suite. Rerun it on every pipeline change — chunk size, embedding model, reranker, prompt.

# Eval set shape (JSONL)
{"question": "What does clause 4.2 cover?",
 "gold_chunk_ids": ["contract-a#chunk-37"],
 "source": "user-logs"}

Tooling: RAGAS and Friends, Locally

from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy, context_precision,
    context_recall, LLMContextRecall
)

# Point RAGAS at a local Ollama endpoint
os.environ["OPENAI_BASE_URL"] = "http://localhost:11434/v1"
os.environ["OPENAI_API_KEY"] = "ollama"

result = evaluate(dataset, metrics=[
    faithfulness, answer_relevancy,
    context_precision, context_recall,
])
print(result.to_pandas())

RAGAS runs against any OpenAI-compatible endpoint, which means Ollama works out of the box. Pair the scores with the manual reranking and hybrid-search experiments to see which change actually moved the numbers — that's the loop that turns a "mostly works" RAG into a measurable one. Also see Top 10 RAG Tools for where evaluation fits the wider ecosystem.

A Minimal Evaluation Workflow

1. Baseline: run your current pipeline over the eval set, record all five metrics.
2. Change one thing (chunk size, embedding model, reranker on/off, prompt).
3. Rerun, compare. If nothing moved, the change doesn't matter to your corpus.
4. Ship the changes that improve metrics without breaking answers you've manually reviewed.
5. Re-run monthly — new documents quietly degrade retrieval over time.

Frequently Asked Questions (FAQ)

What metrics should I use to evaluate RAG?

Split into two layers. Retrieval: recall@k (did the right chunk get retrieved), MRR (how high was it ranked), NDCG@k (ranking quality). Generation: faithfulness (is the answer grounded in the context) and answer relevance (does it answer the question). Track all five; they isolate which layer is failing.

How many eval questions do I need?

Start with 30–50 high-quality questions built from real user queries — enough to catch major regressions. 100+ gives stable scores for small changes like chunk-size tweaks. Quality beats quantity: questions must be realistic and have known-good answers.

What is faithfulness in RAG evaluation?

Faithfulness measures whether every claim in the answer is supported by the retrieved context. A faithful answer can still be wrong or irrelevant — which is why it is paired with answer relevance. Low faithfulness means the model is hallucinating despite good retrieval.

What is a good recall@k score?

For a well-tuned RAG system on its own corpus, recall@5 above 0.8 and recall@10 above 0.9 are reasonable targets. Below 0.7 signals a chunking, embedding, or retrieval problem worth fixing before touching generation.

Do I need an LLM judge to evaluate RAG?

For retrieval metrics, no — they are computed from rankings against labeled chunks. For generation metrics, an LLM judge (or framework like RAGAS) is the standard approach, and a local sub-12B model works for judging in 2026, keeping the whole loop offline.

How do I build an eval set without labeled data?

Two honest paths: (1) mine real questions from logs or from domain experts and label the expected chunk manually; (2) generate synthetic questions from your own chunks with an LLM (RAGAS does this), then spot-check a sample by hand. Never trust synthetic labels without human review.

Sources & Further Reading

• How to Build a RAG System in 30 Minutes (Local, Free)
• Advanced RAG: Chunking Strategies Compared
• Reranking in RAG: Why Retrieval Order Matters
• What Is Hybrid Search? BM25 + Semantic Retrieval
• Top 10 RAG Tools in 2026
• Lawyer Assistant: privacy-first legal RAG
• RAGAS — documentation