RAGAS vs TruLens vs DeepEval: Best RAG Evaluation Framework (2026)
A working comparison of RAGAS 0.2, TruLens 1.x, and DeepEval 2.x for evaluating RAG pipelines: metrics, judge models, CI wiring, and where each one actually fits.
RAGAS, TruLens, and DeepEval are the three dominant open-source frameworks for evaluating Retrieval-Augmented Generation pipelines in 2026, and the right choice depends on whether you need reference-free metrics (RAGAS), tight observability integration (TruLens), or a pytest-style developer workflow (DeepEval). All three implement the standard RAG triad (faithfulness, answer relevance, and context relevance), but they diverge sharply on ground-truth requirements, judge model configurability, CI ergonomics, and how well they scale beyond toy corpora. I've shipped all three across production RAG stacks, and honestly, this guide is the comparison I wish existed before I picked one.
RAGAS 0.2 is the reference implementation of reference-free RAG metrics (faithfulness, context precision, answer relevance) and it integrates cleanly with LangChain, LlamaIndex, and Haystack.
TruLens 1.x wins when you already have observability set up. Its feedback functions attach to spans and stream results into the same trace UI your app writes to.
DeepEval feels like pytest for LLMs: @pytest.mark.parametrize, assertions, and CI hooks come for free, and its G-Eval metric lets you write custom judges in a few lines.
All three now support Claude Sonnet 4.5, GPT-5.1, and Gemini 2.5 Pro as judge models. RAGAS additionally ships a local judge based on Qwen2.5-32B-Instruct.
Ground-truth-free evaluation is table stakes in 2026, but ground-truth-based metrics (context recall, answer correctness) still catch failure modes that judge models routinely miss.
Pick RAGAS for research and benchmarking, TruLens for production monitoring, and DeepEval for CI/CD gating of RAG changes.
What is RAG evaluation and why is it different?
RAG evaluation measures whether a retrieval-augmented pipeline (1) fetches the right context, (2) generates an answer that stays grounded in that context, and (3) actually addresses the user's question. It's different from generic LLM evaluation because a hallucination isn't just "wrong text." It's a specific failure mode where the model invented content the retriever never surfaced. That means you can't evaluate a RAG pipeline the way you'd evaluate a chatbot: BLEU and ROUGE tell you nothing, and even an LLM-as-a-judge scoring only the final answer will miss the case where retrieval returned garbage but the model happened to guess correctly.
The field has largely converged on the RAG triad, popularized by TruLens: context relevance (did we retrieve the right chunks?), groundedness/faithfulness (does the answer only claim things the context supports?), and answer relevance (does the answer address the user's question?). Add context recall and answer correctness when you have ground-truth answers, and you cover most real failure modes. RAGAS, TruLens, and DeepEval all implement variants of these metrics; where they differ is in how you compute them, plug them into your app, and act on the results. My colleague's write-up on LLM-as-a-judge bias and calibration is worth reading alongside this one, because every framework here is a judge model wearing different clothing.
RAGAS vs TruLens vs DeepEval at a glance
Here's the head-to-head across the dimensions I actually care about when picking a framework, not just feature lists, but real production concerns like judge cost, CI ergonomics, and how easy it is to swap in a self-hosted judge.
None of them are "wrong" choices. They aim at different points in the lifecycle. In several stacks I've owned, we ran DeepEval for pre-merge gating and TruLens for live-traffic monitoring at the same time. They don't conflict, and the observability picture is stronger with both.
RAGAS 0.2: reference-free metrics done right
RAGAS ("Retrieval Augmented Generation Assessment") started as a research artifact and grew into the reference implementation of reference-free RAG metrics. The 0.2 line (released Q3 2026) rewrote the metric API around an explicit SingleTurnSample / MultiTurnSample schema and dropped the old evaluate(dataset=...) pandas-heavy pathway in favour of async batching. If you saw examples online using ragas.metrics.faithfulness as a callable, those are from the 0.1 API and will break. The 0.2 RAGAS documentation is the source of truth.
So, let's dive in. Here's a working 0.2 example that evaluates a single RAG interaction with three reference-free metrics plus one ground-truth metric:
from ragas import SingleTurnSample, evaluate
from ragas.dataset_schema import EvaluationDataset
from ragas.metrics import (
Faithfulness,
ResponseRelevancy,
LLMContextPrecisionWithoutReference,
LLMContextRecall,
)
from ragas.llms import LangchainLLMWrapper
from langchain_anthropic import ChatAnthropic
judge = LangchainLLMWrapper(
ChatAnthropic(model="claude-sonnet-4-5", temperature=0)
)
sample = SingleTurnSample(
user_input="What retention policy applies to invoice PDFs?",
response="Invoice PDFs are retained for 7 years per policy FIN-2024-03.",
retrieved_contexts=[
"Policy FIN-2024-03: financial documents including invoices "
"must be retained for seven (7) years from close of fiscal year.",
"Marketing assets follow a 2-year retention window.",
],
reference="Invoices are retained for 7 years under FIN-2024-03.",
)
dataset = EvaluationDataset(samples=[sample])
result = evaluate(
dataset=dataset,
metrics=[
Faithfulness(llm=judge),
ResponseRelevancy(llm=judge),
LLMContextPrecisionWithoutReference(llm=judge),
LLMContextRecall(llm=judge),
],
show_progress=False,
)
print(result.to_pandas())
What I like about RAGAS: the metric decomposition is disciplined. Faithfulness extracts claims from the answer, then asks the judge whether each claim is supported by the retrieved context. You get per-claim provenance, not a black-box score. Context precision iterates the retrieved chunks and asks whether each one was necessary; low precision is a direct signal to raise your reranker threshold or shrink top_k.
What frustrates me: the async concurrency defaults are still too aggressive for Anthropic's tier-1 rate limits, and you'll want to set run_config = RunConfig(max_workers=4) unless you enjoy 429s. I hit this exact bug shipping a batch backfill last quarter, and if retries and backpressure keep tripping you up, the piece on LLM rate limiting and 429 retries covers the patterns that fix it.
TruLens 1.x: feedback functions on live spans
TruLens is the framework I reach for when the goal isn't offline benchmarking, but observability of what's happening right now. Its central abstraction, the feedback function, attaches to spans emitted by your app and computes a metric asynchronously, storing the result alongside the trace. You get the RAG triad in the same UI as your latency histograms, and you can filter traffic by "faithfulness < 0.7" the way you'd filter by HTTP 5xx in a normal APM.
The 1.x rewrite made TruLens OpenTelemetry-native, which means it now composes with whatever tracing backend you already run. Here's a minimal instrumentation that adds the RAG triad to an app already producing OTel spans:
from trulens.core import TruSession, Feedback
from trulens.providers.openai import OpenAI as TruOpenAI
from trulens.apps.custom import TruCustomApp, instrument
provider = TruOpenAI(model_engine="gpt-5.1-mini")
f_groundedness = (
Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness")
.on(context=lambda r: r["retrieved_chunks"])
.on_output()
)
f_answer_relevance = (
Feedback(provider.relevance, name="Answer Relevance")
.on_input()
.on_output()
)
f_context_relevance = (
Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance")
.on_input()
.on(context=lambda r: r["retrieved_chunks"])
.aggregate(lambda scores: sum(scores) / len(scores))
)
class MyRagApp:
@instrument
def query(self, question: str) -> dict:
chunks = self.retriever.search(question, k=5)
answer = self.llm.generate(question, chunks)
return {"answer": answer, "retrieved_chunks": chunks}
session = TruSession()
tru_app = TruCustomApp(
MyRagApp(),
app_name="support-rag",
app_version="v2",
feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
)
with tru_app as recording:
tru_app.app.query("How do I rotate an API key?")
The killer feature is the "app version" concept: TruLens tracks feedback scores per version, so an A/B test between "reranker on" and "reranker off" is a one-line change plus a new version string. The web UI lets you slice by version and compare distributions directly, no notebook required.
Watch out for one gotcha: TruLens's groundedness_measure_with_cot_reasons runs the judge with chain-of-thought reasoning, which triples token cost versus the plain groundedness variant. For high-QPS apps I sample. Five percent of traffic gets the full CoT variant, 95% gets the cheap one, and I reconcile via bootstrap.
DeepEval: pytest for LLM applications
DeepEval is the developer-experience winner. If you've written pytest, you already know how to write DeepEval tests. The framework literally exposes an @pytest.mark.parametrize-compatible decorator and an assert_test function that fails the test when a metric drops below threshold. That means RAG evaluation slots directly into your existing CI pipeline: no separate eval harness, no bespoke dashboards you have to remember to check.
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
FaithfulnessMetric,
AnswerRelevancyMetric,
ContextualPrecisionMetric,
GEval,
)
# Judge configured once via env var: DEEPEVAL_JUDGE_MODEL=claude-sonnet-4-5
@pytest.mark.parametrize("case", [
LLMTestCase(
input="What retention policy applies to invoice PDFs?",
actual_output="Invoice PDFs are retained for 7 years per policy FIN-2024-03.",
expected_output="Invoices are retained for 7 years under FIN-2024-03.",
retrieval_context=[
"Policy FIN-2024-03: financial documents including invoices "
"must be retained for seven (7) years from close of fiscal year.",
],
),
])
def test_rag_pipeline(case):
tone_metric = GEval(
name="Professional Tone",
criteria="Answer must cite the policy identifier and avoid hedging language.",
evaluation_params=["actual_output"],
threshold=0.7,
)
assert_test(case, [
FaithfulnessMetric(threshold=0.85),
AnswerRelevancyMetric(threshold=0.8),
ContextualPrecisionMetric(threshold=0.7),
tone_metric,
])
GEval is DeepEval's escape hatch, a general-purpose LLM-as-a-judge metric you configure by writing a criterion in natural language. It's basically the same idea as OpenAI's G-Eval paper: the judge produces a chain-of-thought rubric, then scores. In practice I use GEval for domain-specific concerns (tone, PII leakage, brand safety) and stick to the built-in metrics for the RAG triad. The built-ins are calibrated against public benchmarks and behave more consistently across judge models.
Faithfulness, context precision, and the RAG triad explained
All three frameworks compute the same headline metrics with subtly different implementations. Here's what each one actually measures, in plain terms, so you can debug scores that look wrong.
Faithfulness (a.k.a. groundedness)
The judge extracts atomic claims from the generated answer, then checks each claim against the retrieved context. The score is the fraction of claims that are supported. RAGAS decomposes claims into short statements; TruLens uses a single-shot NLI-style prompt; DeepEval uses a hybrid. A faithfulness score below 0.7 almost always means the model is speculating beyond what retrieval provided.
Answer relevance
The judge generates N hypothetical questions the answer could plausibly be answering, then measures cosine similarity between those questions and the user's actual query. Low answer relevance means the model is verbose, off-topic, or dodging the question.
Context precision and context recall
Precision asks: "of the chunks we retrieved, which ones were actually necessary?" Recall asks: "of the chunks we should have retrieved (per ground truth), how many did we?" Precision is reference-free; recall requires ground truth. Chunks retrieved but never cited hurt precision; missing chunks hurt recall. My reranking cheat sheet in Cohere Rerank vs Voyage vs Jina vs BGE maps directly to fixes for precision problems.
Answer correctness (ground-truth)
A hybrid of semantic similarity between the generated answer and the reference plus an F1 over extracted factual claims. This is the metric most correlated with human ratings in the RAGAS paper, but you can only compute it when you have ground-truth answers.
How do you pick a judge model for RAG evaluation?
Judge choice matters more than framework choice. All three tools defer the actual scoring to an LLM, so a judge that's biased, over-confident, or poorly calibrated will make even the best framework produce misleading dashboards. My current defaults, for what they're worth:
Claude Sonnet 4.5 as the primary judge for faithfulness and context precision. It's the least prone to false-positive "supports" that I've measured, and its long-context handling is stable up to 200K tokens.
GPT-5.1 for answer relevance and G-Eval-style custom criteria. Slightly better at following elaborate rubrics; also cheaper for short-context evals.
A local Qwen2.5-32B-Instruct (via vLLM) for offline batch eval where scale matters more than per-sample accuracy. RAGAS ships this out of the box; DeepEval works with it via LiteLLM.
Whichever you pick, calibrate. Take 50 to 100 samples, have a human label them, then measure the judge's Cohen's kappa against the human labels. Anything below 0.6, and your dashboards are theatre. Repeat this once per quarter and every time you swap the judge model. I've been burned twice by silently degraded scores after a "harmless" judge upgrade.
Wiring RAG evaluation into CI/CD
Offline eval that only runs in a Jupyter notebook once a month is useless. The whole point of scoring is catching regressions before they hit users, and that means the eval must run on every PR that touches a retriever, an embedding model, a chunker, or a prompt. Here's the pattern that has worked for me:
Maintain a golden test set of 100 to 300 curated (question, context, reference) triples, versioned in the repo alongside the code.
Wrap eval in DeepEval (or DeepEval's deepeval test run CLI, which shells to pytest under the hood).
On every PR, run against the golden set with strict thresholds: faithfulness > 0.85, answer relevance > 0.8, context recall > 0.75. Failures block merge.
Nightly, run a broader "silver" set of 2K to 5K samples with softer thresholds and post the trend to Slack.
In production, sample 1 to 5% of live traffic into TruLens for the same metrics, using the same judge model, so offline and online numbers are comparable.
For durable execution of the nightly job (retries, idempotency, cost caps), I usually run it on Inngest or Trigger.dev; the trade-offs are covered in the Inngest vs Trigger.dev vs Hatchet comparison. The DeepEval docs also cover a hosted alternative via Confident AI, which is worth considering if you don't want to manage the harness yourself.
Common pitfalls that make RAG scores lie
Every team I've worked with has hit at least three of these. Skimming this list first is cheaper than debugging a green dashboard that hides a broken retriever.
Micro test sets. Ten hand-picked questions won't detect a 3% regression. Aim for a golden set large enough that a real degradation shows up outside noise: 100 minimum, 300 comfortable.
Judge bias for length. Judges systematically prefer longer answers. If your model change increases verbosity, "answer relevance" will go up even if quality didn't. Bake a length-normalized variant into your dashboard.
Retrieval leakage in synthetic questions. Auto-generated test sets often contain lexical overlap with source chunks so severe that BM25 alone gets perfect recall. Score your synthetic set with a simple retriever first; if it aces the test, the set is too easy.
Ignoring context-precision when you have a reranker. Rerankers optimise for placing relevant chunks first, but if you're still passing 20 chunks into the prompt, precision stays low. Trim top_k after reranking, not before.
Judge model drift. "gpt-4o" today isn't the same model as "gpt-4o" six months ago. Pin exact snapshot IDs (e.g. gpt-5.1-2026-06-01) in your eval config, or your longitudinal comparisons are meaningless.
Correlating eval scores with user satisfaction. Do this once. Take 500 samples with real user thumbs-up/down, run your framework on them, compute Spearman correlation. If it's below 0.4, your metric set doesn't reflect what users care about, and you need to add or remove metrics.
Which framework should you actually pick?
My decision tree, in order of the question I ask myself first:
Are you gating merges on RAG quality? Go with DeepEval. The pytest ergonomics are worth it, and CI is the highest-value place to run eval.
Do you want live scores on production traffic in the same UI as your traces? Go with TruLens. The feedback-function-on-a-span model is unmatched, and OTel-native means no vendor lock-in.
Are you writing a paper, benchmarking retrievers, or comparing embedding models? Go with RAGAS. Its metrics are the ones most cited in the literature, and its synthetic test-set generator, caveats aside, is the most sophisticated.
Doing all three? Run DeepEval in CI + TruLens in prod. They compose. RAGAS optionally as a third opinion for research work.
If you're just starting out and want the shortest path to any signal at all, install DeepEval, write ten test cases, and run deepeval test run. You'll have a working eval pipeline in an afternoon, and you can always graduate to the multi-framework setup once you know what you actually care about measuring. For a broader map of where evaluation sits in the LLM lifecycle, the piece on LLM evaluation for production zooms out beyond RAG-specific concerns. And if you're pairing eval with tracing, the Langfuse vs LangSmith vs Helicone vs Arize Phoenix comparison covers the observability side.
Frequently Asked Questions
Is RAGAS still maintained in 2026?
Yes. RAGAS 0.2.15 shipped in August 2026 and the project remains actively developed by Exploding Gradients. The 0.2 line is a breaking rewrite from 0.1, so audit tutorials for API compatibility before copy-pasting older examples.
Can you evaluate a RAG pipeline without ground-truth answers?
Yes. All three frameworks support reference-free metrics. Faithfulness, answer relevance, and context precision can be computed with only the question, retrieved context, and generated answer. Ground-truth is only required for context recall and answer correctness.
What's the difference between DeepEval and RAGAS?
RAGAS is a metrics library optimized for offline benchmarking; DeepEval is a pytest-shaped test runner optimized for CI/CD. RAGAS's metric implementations are more research-oriented and better documented; DeepEval's developer ergonomics are far superior for regression testing. Many teams use both.
How much does LLM-as-a-judge evaluation cost?
Rough numbers for a 200-sample test set with the RAG triad using Claude Sonnet 4.5 as judge: about $0.30 to $0.80 per full evaluation run in 2026. Chain-of-thought variants (like TruLens's _with_cot_reasons metrics) roughly triple that. Batch API pricing halves it. Local judges cost only compute.
Which judge model gives the most reliable RAG evaluation scores?
In my calibration experiments across 2026, Claude Sonnet 4.5 has the highest agreement with human labels on faithfulness and context precision (Cohen's kappa around 0.72), GPT-5.1 leads on custom rubric-based metrics, and Qwen2.5-32B is the best local option when cost or data-residency matters more than per-sample accuracy.
Benchmarks and drop-in code for the four rerankers that actually matter in 2026: Cohere Rerank 3.5, Voyage Rerank-2, Jina Reranker v2, and BGE v2-M3. Pick the right one for your RAG pipeline.
Compare the four leading LLM observability platforms (Langfuse, LangSmith, Helicone, and Arize Phoenix) with real production patterns for tracing, prompts, and evals.
Compare Claude Extended Thinking, OpenAI o3, and Gemini 2.5 Thinking in real production workloads. Pricing per solved problem, tool-use behavior, latency, and the eval loop I use before shipping any reasoning model to real users.
