How to Evaluate a RAG System Before You Trust It

Why "It Looks Right" Isn't Enough

RAG systems have a failure profile that hides well. The generator produces fluent, confident prose. The citations come from your real documents. Yet studies of grounded-answer systems keep finding the same two failure classes: retrieval misses (the right document was never fetched, so the answer is built on nothing) and faithfulness drift (the model added, contradicted, or guessed beyond what the source said). Neither shows up in a five-question demo.

That's why the industry converged on component-level evaluation: score retrieval and generation separately, so a failure tells you where to fix it. Open-source frameworks like RAGAS define the standard metric families — faithfulness, answer relevancy, context precision, and context recall — and the widely used "RAG Triad" (context relevance, faithfulness, answer relevance) gives the same diagnostic power in three numbers.

Step 1: Build the Test Set Before You Tune Anything

Evaluation is only as honest as its test set. A test set built from the questions you wish users asked will flatter any system. Build one from reality:

1. Mine real questions. Support tickets, past emails, FAQ traffic, or the questions your team actually answers.
2. Cover the document types. If your corpus has contracts, policies, and emails, your test set needs all three — retrieval behaves differently per type.
3. Write the expected answer or the source answer. For each question, record which document(s) contain the answer and what the correct answer is.
4. Include hard cases deliberately: questions whose answer spans two documents, near-duplicate documents, and questions with no answer in the corpus (the system should say "I don't know").

Start with 30–50 questions. That's enough to expose systematic failures, and small enough to hand-check. Grow it continuously: after launch, log real user questions and add the ones the system got wrong — an eval set that only grows with failures gets sharper over time instead of going stale.

Step 2: Run the RAG Triad — and Let It Point the Finger

The RAG Triad scores three things, and each one isolates a different stage of the pipeline:

This is the triad's superpower: it converts "the system is wrong" into "the system retrieves badly" or "the system drifts from its sources." You fix those with completely different tools. (The exact metric formulas — recall@k, MRR, NDCG for retrieval, faithfulness and answer relevance for generation — are covered in the companion post RAG Evaluation: How to Measure Retrieval Quality.)

Step 3: Set Thresholds — and Know What "Good Enough" Means

Scores without thresholds are decoration. Set explicit bars per metric before you measure, based on the risk of getting it wrong:

• Faithfulness ≥ 0.8 (on a 0–1 scale) is a reasonable entry bar for production, and the bar should be higher in regulated domains — a hallucination in a contract summary or a medical answer isn't a rounding error.
• Answer relevance and context relevance ≥ 0.7–0.8 depending on how forgiving the domain is to "close but not quite."
• Zero tolerance for answered-but-unsupported: the "I don't know" case must work. A system that confidently fabricates when the answer isn't in the corpus is not fixable by tuning — it needs a refusal strategy (see Why RAG Hallucinates and How to Fix It).

Step 4: Turn It Into a Loop, Not a Certification

An evaluation you run once is a snapshot of last month's system. Every change — new model, new chunk size, new embedding, new prompt — can silently move scores. The trustworthy setup keeps measuring:

1. Baseline. Run the full test set, record the three triad scores. This is your reference point.
2. Change one thing. Never tune retrieval, prompt, and model in the same iteration — you won't know what moved the score.
3. Re-run, compare, keep or revert. Only keep changes that hold or improve the triad.
4. After launch, keep logging. Track citations clicked, thumbs-downs, and regenerated answers. New question patterns flow back into the test set (Step 1), which keeps the eval sharp.

Automation helps here: evaluation frameworks can score a test set headlessly on every change, so regressions get caught in minutes instead of weeks. But start manual — a spreadsheet of 40 questions, hand-scored, will teach you more about your system than any framework.

The Pre-Trust Checklist

Frequently Asked Questions (FAQ)

How do you evaluate a RAG system?

You evaluate a RAG system on three layers: retrieval (did it find the right documents?), grounding (is the answer faithful to what was retrieved?), and relevance (does the answer actually address the question?). In practice this means building a small test set of real questions with expected answers, running the system against it, scoring each layer, and comparing scores before and after changes.

What is the RAG Triad?

The RAG Triad is a diagnostic framework of three metrics: context relevance (did retrieval return useful documents?), faithfulness (does the answer stick to those documents, with no hallucination?), and answer relevance (does the answer actually answer the question?). The triad pinpoints which stage of the pipeline is failing instead of just reporting a single accuracy number.

How many test questions do I need to evaluate a RAG system?

Start small: 30–50 real questions covering the main document types and question patterns your users will ask. That is enough to expose systematic failures. Grow the set over time by logging real user questions and adding the ones the system got wrong — an evaluation set that only contains easy questions will make the system look better than it is.

What is a good faithfulness score?

Treat 0.8 or higher on a 0–1 faithfulness scale as the entry bar for production, because faithfulness failures are hallucinations — the most dangerous error class in legal, medical, and financial answers. Relevance targets depend on your domain, but a system with high faithfulness and low relevance usually has a retrieval problem, not a generation problem.

When is a RAG system ready to trust?

When three things hold: it scores above your thresholds on a representative test set, the failures you find are acceptable and documented (not surprises), and you have a way to keep measuring after launch by logging questions, citations, and user feedback. Evaluation is a process, not a one-time certification.