Evaluating agentic systems

The unit of evaluation here is the trajectory — the whole sequence of tool calls, arguments and observations that led to an answer. Not the answer.

That is the entire guide in one sentence, and almost every eval tool will let you ignore it. Score an agent the way you score a chatbot and you will measure whether the last message was correct, which is a fact about luck as much as about capability.

The run that passes and should not

Here is a real failure shape. The agent is asked to fix a failing test. It succeeds. The final tests pass. Response-level scoring records a win.

The run cost four wasted tool calls, three of which would have failed the same way forever if the loop had not broken by chance. Ship this agent and it will hit the cases where that chance does not arrive: the loop continues, the budget runs out, the user waits.

An agent score is a system score

This point is worth stating flatly, because the entire leaderboard on this site depends on it too: an agentic score measures the model and its harness together.

Swap the scaffold — different tool definitions, a different retry policy, a different system prompt, a different context window budget — and the number moves, sometimes by more than the gap between two models. Benchwiki's own records make this point repeatedly, and it is why this site's agentic tab keys its rows on model plus harness and shows two rows for the same model under two scaffolds.

Look at what that means in practice on a benchmark you have probably seen quoted:

When someone says "this model scores X on SWE-bench", the honest question is "under which harness?" Without that, the number is not reproducible and not comparable — and the second of those two benchmarks is already saturated, so the number is telling you less than it appears to regardless.

What actually breaks

Five failure shapes, in rough order of how often they cost real money:

1. Wrong tool. The agent reaches for search when it needed list. Usually a tool description problem, not a model problem.
2. Right tool, wrong arguments. Far more common and much harder to spot, because the call succeeds and returns something plausible.
3. Loops. The same call with the same arguments, repeatedly. Cheap to detect, and almost nobody detects it.
4. Non-termination. The agent neither finishes nor fails. This is the one that quietly consumes a budget.
5. Accidental recovery. The run above. The worst of the five, because it looks like success in every response-level metric you have.

Measures that catch them

Score tool selection separately from argument construction. They have different fixes: selection problems are usually solved by rewriting tool descriptions, argument problems by constraining schemas or adding validation. A blended "tool use" score tells you something is wrong and nothing about which.

Three ways to match a trajectory, and when each is right

This is where most trajectory evaluation goes wrong, so be deliberate about it.

Exact match requires the same calls in the same order. Use it only when the order is genuinely load-bearing — a migration that must back up before it writes, a transaction that must authorise before it charges. Anywhere else it punishes valid strategies.

Any-order match requires the same set of calls, order irrelevant. This is the right default for read-only information gathering, where fetching the schema before the rows or after is a matter of taste.

Superset match requires that the reference calls all appear, and tolerates extra ones. Use it when you care that the agent did the necessary things and you are not yet trying to make it efficient. Pair it with a step-count budget, or you have licensed the forty-step run.

A worked example

This runs. It needs no framework — the point is that trajectory scoring is not exotic machinery, it is a few lines you can write today and then adopt a framework for later.

"""Score an agent run on its trajectory, not its final answer.

Run: python trajectory_eval.py
"""
from dataclasses import dataclass, field


@dataclass
class Call:
    tool: str
    args: dict


@dataclass
class Run:
    calls: list[Call]
    final_state_correct: bool
    cost_usd: float = 0.0


@dataclass
class Verdict:
    passed: bool
    reasons: list[str] = field(default_factory=list)


def score(run: Run, required: list[str], step_budget: int) -> Verdict:
    """Any-order superset match, plus the three checks response scoring cannot make."""
    reasons: list[str] = []
    used = [call.tool for call in run.calls]

    # 1. Did it do the necessary things at all?
    missing = [tool for tool in required if tool not in used]
    if missing:
        reasons.append(f"never called: {', '.join(missing)}")

    # 2. Loops: the same call with identical arguments, back to back.
    #    Note "back to back". Flagging *any* repeat is the obvious first implementation and
    #    it is wrong: re-running the tests after a fix repeats an earlier call legitimately.
    previous: tuple[str, str] | None = None
    for call in run.calls:
        key = (call.tool, repr(sorted(call.args.items())))
        if key == previous:
            reasons.append(f"repeated identical call: {call.tool}({call.args})")
        previous = key

    # 3. Efficiency: a correct answer after forty steps is not the same as after four.
    if len(run.calls) > step_budget:
        reasons.append(f"took {len(run.calls)} steps against a budget of {step_budget}")

    # 4. And only now, the thing response-level scoring would have checked alone.
    if not run.final_state_correct:
        reasons.append("final state is wrong")

    return Verdict(passed=not reasons, reasons=reasons)


if __name__ == "__main__":
    lucky_recovery = Run(
        calls=[
            Call("search_docs", {"q": "failing test"}),
            Call("search_docs", {"q": "failing test"}),      # loop
            Call("read_file", {"path": "src/mian.py"}),       # typo, does not exist
            Call("read_file", {"path": "src/mian.py"}),       # loop
            Call("run_tests", {}),                            # before any fix
            Call("list_dir", {"path": "src"}),
            Call("edit_file", {"path": "src/main.py"}),
            Call("run_tests", {}),
        ],
        final_state_correct=True,
    )

    verdict = score(lucky_recovery, required=["list_dir", "edit_file"], step_budget=5)
    print("response-level scoring would say: PASS")
    print(f"trajectory scoring says: {'PASS' if verdict.passed else 'FAIL'}")
    for reason in verdict.reasons:
        print(f"  - {reason}")

Running it prints:

response-level scoring would say: PASS
trajectory scoring says: FAIL
  - repeated identical call: search_docs({'q': 'failing test'})
  - repeated identical call: read_file({'path': 'src/mian.py'})
  - took 8 steps against a budget of 5

Two lines of difference, one of which is the difference between shipping this agent and not.

Reading the results

• Completion high, steps high. The agent works and wastes money. Look at tool descriptions before you look at the model.
• Completion high, loops present. You are one unlucky run from non-termination. Loop detection belongs in the agent, not only in the eval.
• Tool selection high, argument correctness low. A schema problem. Constrain the arguments or validate before dispatch; a bigger model will not reliably fix this.
• Everything high offline, complaints online. Your golden set does not contain the shapes production sends. Sample real trajectories and add them.

What to take away

Score the path. Score selection apart from arguments. Put a step budget beside every completion number. And whenever you see an agentic score quoted anywhere — including on this site — ask which harness produced it, because the number belongs to the system, not to the model.

Where to go next

The framework comparison covers which tools make trajectory evaluation practical at scale, and the RAG guide covers the other archetype where the unit of evaluation is not a response.