Framework comparisons usually list features in a grid and let you count ticks. That grid is wrong within a quarter, and it answers a question nobody has. The question people actually have is: my application has this shape — what should I reach for?
So this page is organised by archetype, and every claim on it carries a date.
The whole field, by what it serves
| Framework | Serves | Strength | Watch out for | Licence |
|---|---|---|---|---|
| RAGAS | rag | Scores the retriever and the generator separately, which is the distinction RAG evaluation lives or dies on. | RAG-shaped by design. Bending it around an agent or a multi-turn session means reimplementing most of what it does. | Apache-2.0 |
| DeepEval | generation, rag, agentic, multi-turn, extraction | The broadest metric library, driven from a pytest-style workflow, so evals run where your other tests run and fail a build the same way. | Breadth invites using a metric because it exists. Most of its judge-based metrics inherit the judge model's biases. | Apache-2.0 |
| LangSmith | agentic, multi-turn, generation, rag | Tracing first: you can see the trajectory a run actually took, which is the precondition for scoring one. | Hosted and commercial. The data you are tracing is usually the data you are least free to send elsewhere. | Proprietary (hosted) |
| Langfuse | agentic, multi-turn, generation, rag | Open-source tracing and eval with a self-hosted path, which matters once traces contain production data. | Self-hosting is real operational work — it is a database and a service, not a library. | MIT (core) |
| Arize Phoenix | rag, agentic, generation | Strong at retrieval debugging: embedding drift and retrieval quality analysis rather than just a pass or fail number. | Most useful when you already have volume. On a handful of traces it tells you little. | Elastic-2.0 |
| Opik | agentic, generation, rag | Tracing plus evaluation with an open-source core, aimed at moving offline evals online. | Younger than the alternatives; expect faster API movement. | Apache-2.0 |
| promptfoo | generation, extraction, reference | Declarative test matrices across prompts, models and providers, plus red-teaming and jailbreak probes. Runs in CI without a service. | Built around inputs and outputs. It has little to say about a trajectory. | MIT |
| Inspect | reference, agentic | Built by the UK AI Safety Institute for model evaluation, with solvers, scorers and a log viewer designed for auditable runs. | Aimed at evaluating models, not your application. The two are different disciplines. | MIT |
| Evidently | generation, reference | Comes from the classic ML monitoring lineage, so drift and data-quality reports are first-class rather than bolted on. | Its LLM support is newer than its tabular support, and it shows in places. | Apache-2.0 |
| MLflow | generation, reference | If your organisation already tracks experiments in MLflow, its LLM evaluation support puts eval runs beside everything else rather than in a separate tool. | A general experiment tracker with eval features, not an eval framework. The metric library is thin by comparison. | Apache-2.0 |
Organised by which archetype each framework serves, because that is the decision a reader is actually making. Claims here describe what each project is built around, not a feature checklist — checklists go stale faster than they can be maintained. Last verified 2026-09-12
Choosing by shape
You are building RAG. Start with RAGAS. Its whole design is the retriever/generator split that the RAG guide argues for, and you will not have to construct that split yourself. Add Phoenix when you have enough traffic for retrieval debugging to pay off.
You are building an agent. You need tracing before you need metrics, because you cannot score a trajectory you cannot see. LangSmith or Langfuse first — Langfuse if the traces contain data you would rather self-host — then DeepEval or Inspect for the scoring described in the agentic guide.
You are building plain generation or a chat feature. DeepEval in CI, pytest-style, and promptfoo when you need to sweep prompts and models against each other rather than score one pipeline.
You are doing structured extraction. You probably need less than you think: per-field precision and recall against references, plus a schema-validity rate. promptfoo's matrices fit this well, and a plain test suite fits it even better.
You are red-teaming. promptfoo for jailbreak and injection probes; Inspect if you are evaluating models rather than an application.
You already live in an MLOps stack. Evidently or MLflow keep eval results beside the rest of your experiments. Neither has the metric depth of the specialists, and that is often an acceptable trade against having one fewer tool.
The choice that matters more than the tool
Two decisions outrank framework selection, and both are yours regardless of what you pick:
Where evals run. A framework that only runs on your laptop will stop being run. Put the offline suite in CI where a regression fails a build, and put sampling in production where online evals can see the traffic the golden set never had.
Who owns the golden set. The set decides what "better" means. If nobody owns it, it ossifies, and you spend a year optimising against questions that stopped being representative months ago. Review it on a schedule and record when you last did — the same discipline this page applies to itself.
A note on lock-in
Traces are the sticky part, not the metrics. Metric implementations are a few dozen lines each and you can move them; a year of production traces in a hosted schema is a migration. If that worries you, weigh the self-hostable options — Langfuse, Phoenix, Opik — before the volume makes the decision for you.