Evals
Sections 1 and 2 of this site are about model evaluation: ranking foundation models against shared benchmarks. This section is about application evaluation: whether your system works. RAGAS and DeepEval will not tell you whether Claude beats Gemini. They tell you whether your pipeline hallucinates. If you arrived here from the leaderboard expecting more rankings, this is a different discipline — and the more useful one once you have chosen a model.
The spine of everything below is the unit of evaluation. Most eval tooling assumes one input, one output, one score. Real applications are not that shape: a RAG answer is a response plus its retrieved context, and an agent run is a whole trajectory. Score the wrong unit and your numbers are confidently meaningless.
Start here
- Foundations
Why one eval pipeline is never enough, and why the unit you score decides everything downstream.
Unit of evaluation: varies by archetype · intro · Last verified 2026-09-12
Guides
- Evaluating agentic systems
Scoring the path an agent took, not just where it ended up — and why an agent score is a system score.
Unit of evaluation: the whole trajectory · advanced · Last verified 2026-09-12
- Which eval framework, for which archetype
Choosing an eval framework by the shape of your application rather than by feature checklist.
Unit of evaluation: varies by archetype · intermediate · Last verified 2026-09-12
- Evaluating RAG and knowledge assistants
Scoring the retriever and the generator separately, because they fail independently and their fixes are opposite.
Unit of evaluation: one response plus its retrieved context · intermediate · Last verified 2026-09-12
Still to be written
One guide per archetype is the plan: multi-turn assistants, structured extraction, code generation, multimodal and document systems, and voice pipelines. They are not here yet. Listing them as missing is more useful than a page that pretends the set is complete.
Framework claims on this site are dated. Last verified 2026-09-12