RAG evaluation metrics: separate retrieval from answer quality
A correct answer can hide bad retrieval, and a faithful answer can repeat an outdated document. Score each stage before changing the model.
A RAG pipeline can retrieve the wrong document, omit a necessary passage, misread the right evidence, or answer correctly from the model's prior knowledge. One aggregate accuracy score cannot tell you which happened.
Evaluate retrieval and generation separately, then measure the outcome of the complete pipeline. The useful question is which change would fix the failed case: indexing, retrieval, context construction, or answer generation.
Keep the evidence for every example
Store the question, retrieved passages in order, document identifiers and versions, final context supplied to the model, generated answer, and reference answer where one exists. Preserve retrieval scores and filters as diagnostic metadata.
The retrieved set and the final context are not necessarily identical. Reranking, truncation, deduplication, and formatting can remove the passage the generator needs. A trace that only records the initial search results leaves that failure invisible.
Use a small set of distinct metrics
| Metric | What it asks | Evidence needed |
|---|---|---|
| Precision at k | How many of the top k retrieved items are relevant? | Relevance judgments for returned items |
| Recall at k | How many known relevant items appear in the top k? | A reference set of relevant items |
| Ranking quality | Are the most useful items near the top? | Relevance labels, optionally graded |
| Faithfulness | Are the answer's claims supported by supplied context? | Answer and final context |
| Answer correctness | Does the answer agree with the reference facts? | Answer and a trustworthy reference |
| Task completion | Did the user receive the required result? | A task-specific rubric or outcome |
Metric names are not interchangeable across libraries. Ragas's Context Precision is sensitive to the ordering of useful chunks; it is not simply the fraction of relevant chunks returned. Read the implementation before comparing its value with a hand-written precision-at-k scorer. Ragas Context Precision definition.
Ragas also separates faithfulness, response relevancy, and retrieval metrics. These measure different aspects of a RAG response; they should not be treated as interchangeable accuracy scores. Ragas metric catalog.
Diagnose a failed answer
Suppose a user asks for the return window for a product. The current policy says 30 days, but the system answers 60 days. This illustrative case has several possible explanations:
- The index contains only the old 60-day policy. Fixing the prompt cannot supply the missing current document.
- The current policy is indexed but ranks below unrelated results. Inspect filters, query rewriting, chunking, and ranking.
- Retrieval returns the current policy, but context truncation removes it. Inspect the context-construction step.
- The model receives the current policy and still answers 60 days. Inspect generation and the instruction to resolve conflicting evidence.
If the supplied context contains only the old policy, the answer may be faithful and factually outdated at the same time. Grounding in retrieved text does not establish that the text is current or authoritative.
Compare changes on the same cases
To test a retriever change, hold the generator and scoring configuration fixed. To test a generator change, reuse the same stored contexts where possible. Run the full pipeline as a separate experiment so improvements in one stage do not conceal regressions elsewhere.
Report scores by question type: direct lookup, multiple-document synthesis, unanswerable questions, and conflicting sources. Include the number of cases in each group. For unanswerable questions, score appropriate abstention rather than rewarding an answer merely because it sounds complete.
Inspect examples that improved and examples that regressed. An average can rise while a small but important workflow gets worse. Keep latency and cost beside quality rather than blending them into an unexplained composite score.
Choose tools around the diagnostic workflow
Ragas is worth examining when you need explicit retrieval and generation metrics. An experiment workspace such as Braintrust can organize versioned test cases and results; your instrumentation and scorers still need to preserve the stage boundaries. Braintrust dataset documentation.
See the RAG evaluation tools comparison for the broader shortlist. Before trusting any model-based metric, calibrate its judgments against examples your team has reviewed.
Continue with the evals implementation guide for the full workflow.