RAG evaluation: Ragas vs DeepEval vs TruLens vs LangSmith
A single score cannot explain whether a retrieval-augmented generation (RAG) system works. It may fail while parsing documents, splitting them into chunks, retrieving or reranking evidence, generating an answer, attaching citations, or applying filters. Measure each stage separately so a regression points to a specific part of the pipeline.
Start with retrieval metrics on a small labeled dataset. Choose one tool for the features your existing test runner lacks. Ragas provides RAG metrics, DeepEval provides test-style checks, and TruLens attaches feedback to individual runs. LangSmith is another option when you want traces and datasets together. Add custom checks for product-specific failures.
Choose by required metrics, test-runner integration, trace review, and where evaluation data may be processed.
Decision table
| Need | Starting point | Why |
|---|---|---|
| Cheap retrieval regression checks | Local metrics | Recall@k, MRR, nDCG, filter false-exclusion, and citation ID/span validity can use fixed labels or rules. |
| RAG metric library | Ragas | Provides context precision, context recall, response relevancy, faithfulness, and related metrics. |
| CI gates for LLM applications | DeepEval | Test-case style interface works well when evals should fail a PR or deployment. |
| Explainable app feedback | TruLens | RAG triad separates context relevance, groundedness, and answer relevance. |
| Trace-centered product evals | LangSmith | Datasets, evaluators, annotations, traces, and regression workflows live together. |
| Domain-specific quality | Custom evals | Generic metrics rarely know your ontology, filters, citation policy, parser constraints, or refusal rules. |
Metrics by pipeline stage
| Stage | First metrics to add | Why |
|---|---|---|
| Parsing | extraction completeness, table preservation, page coverage | A retriever cannot return text that parsing omitted. |
| Chunking | chunk answerability, boundary loss, duplicate rate | Splitting related facts can prevent retrieval of enough context. |
| Retrieval | Recall@k, MRR, nDCG@k, context precision, context recall | Check which expected sources reach the answer model. |
| Reranking | Precision@1, nDCG change, added latency | Retain a reranker when better ordering justifies the added runtime. |
| Generation | faithfulness, groundedness, answer relevance | Faithfulness checks context support; answer relevance checks whether the response addresses the question. |
| Citations | claim coverage, citation support, unsupported claim rate | A grounded answer without useful citations can still fail the product. |
| Production | fallback rate, correction rate, p95 latency, cost per answer | Offline quality is incomplete without operational telemetry. |
Tool notes
Recall@k measures how many labeled relevant items appear in the first k results. MRR measures the rank of the first relevant item; nDCG also accounts for different relevance levels. These retrieval metrics need relevance labels. An LLM judge that finds an answer consistent with context cannot replace them.
Ragas provides context precision, context recall, faithfulness, and response relevance metrics. Its context recall requires a reference answer or reference contexts. Context precision has variants with and without references. Faithfulness checks the response against retrieved context; it does not need a reference answer. Check each metric’s required inputs before preparing a dataset. Calibrate model judges against reviewed examples and record their prompts, model versions, and cost.
DeepEval fits engineering workflows where evaluation should behave like tests. It is useful for CI regression checks, especially around known failure cases. The caution is that test-style evals are only as good as the cases you maintain.
TruLens works well when you want feedback functions tied to app records. The RAG triad is useful because it keeps context relevance, groundedness, and answer relevance separate instead of compressing them into one opaque number.
LangSmith is practical when your traces, runs, datasets, and review workflow are already in the LangChain/LangGraph ecosystem. It also supports other frameworks. A local runner may be simpler if all evaluation data and execution must stay on your machine.
Add custom checks for constraints the generic metrics do not cover. If your RAG system filters documents by permission, jurisdiction, date, product line, or ontology, measure false exclusions and policy mistakes directly.
A reasonable first stack
- Build a golden set that covers important query slices and expected source IDs. A few dozen cases can expose early regressions; expand until confidence intervals and failure coverage support the release decision.
- Track deterministic retrieval metrics locally before adding LLM judges.
- Choose one library or platform if you need a groundedness judge, test integration, or trace review that your runner lacks.
- Use it with custom checks for filters, citations, parser quality, and refusal behavior.
- Save results and review samples. Add another tool only when you can name a missing capability.
Checking that a citation ID exists or that a quoted span occurs is a deterministic test. It does not prove that the source supports the claim. For that, use reviewed claim-source labels or a calibrated semantic judge. As the TruLens RAG triad distinguishes, an answer can address the question yet lack support in the supplied context.
A common mistake
Do not stop after measuring faithfulness. Faithfulness asks one narrow question: does the answer agree with the retrieved context? It cannot tell you whether the retriever found the right source. It also misses lost tables, incorrect permission filters, and citations that point to the wrong passage.
Deeper reading
- RAG Evaluation Metrics gives the full stage-by-stage framework.
- Search Ranking Stack covers retrieval and reranking design.
- Context Engineering for AI Agents explains why context assembly is part of the system, not prompt decoration.