Two runs of the same coding agent on the same task. Both end green.

The first one read three files, found the bug, wrote a four-line fix, ran the tests, and stopped. Twenty-two thousand tokens. The second one read forty-one files, tried an approach, abandoned it, tried a second approach, hit a failing assertion, edited the assertion, re-ran, went green, and reported success. Three hundred thousand tokens.

Your eval reports 2/2.

That's the whole problem with outcome-only evaluation, stated as plainly as I can put it. A pass/fail on the final state is one bit of information about a process that produced thousands. It tells you something happened. It cannot tell you whether the thing that happened is something you want more of.

Outcome evals are necessary and almost useless

I want to be careful here, because the reaction to this argument is usually an overcorrection. Outcome evals are not bad. They are the only part of your eval stack that is cheap, objective, and automatable, and if you don't have them you have nothing. Run them first.

They're also blunt in a specific way: they tell you that something went wrong without telling you where or why. On a five-step task that's fine, because you can just read the whole thing. On a sixty-turn agentic task with a hundred tool calls, "failed" is not a diagnosis. You have a red mark and no idea whether the model misunderstood the request, picked the wrong tool, got a useless error message back, or ran out of context halfway.

And in the other direction, a pass tells you even less. A correct final answer reached in twenty steps with two policy-violating intermediate calls is a failing trajectory that your outcome grader records as a win.

So you need a second axis. The final state is one score. The path is another. Score them independently, because they come apart in both directions and both directions are informative:

High reasoning, low performance. The agent decomposed the problem correctly, formed the right hypothesis, and then ran out of turns or hit a real environment limitation. The approach was right. Your budget or your tooling was wrong. This is good news wearing a red badge.

Low reasoning, high performance. The agent guessed, brute-forced, pattern-matched something memorized, or fit the test. It passed. If you are generating training data, this is the dangerous quadrant, because your outcome grader says reward this and the model learns exactly the wrong lesson.

That second case is the reason trajectory evaluation exists.

The six things worth scoring on the path

You can invent your own dimensions and most teams do, but the ones that have converged across eval tooling in the last year are roughly these. I score each independently, binary where possible, with a written note.

Tool selection. Did it reach for the right tool? The classic failure is using a general-purpose shell call for something a purpose-built tool does better, usually because the tool descriptions overlap and the model picked the first plausible one.

Argument construction. Right tool, wrong arguments. A malformed path, a filter that excludes the thing it's looking for, a regex that doesn't compile. This is where a surprising share of "the model is dumb" complaints actually live.

Result utilisation. It called the tool, it got a result, and then it ignored the result. Watch for the agent reading a file and then writing code that contradicts what it just read.

Error recovery. Something failed. What happened next? Good recovery reads the error, updates the plan, and tries a different thing. Bad recovery runs the identical command again.

Plan coherence. Do the steps compose into the goal? An agent can execute every individual step competently while the sequence goes nowhere.

Task completion. Did it do all of what was asked, or the first third of it and then declare victory? Under-delivery followed by a confident summary is one of the most common failures in long-horizon runs and one of the easiest to miss, because the summary reads well.

The reason to keep these separate rather than collapsing them into one "trajectory score" is that the mix tells you what to fix. Four failures concentrated in argument construction is a tool-schema problem. Four failures in error recovery is an error-surfacing problem. One aggregate number tells you neither.

Finding the turn where it went wrong

Scoring dimensions gives you a profile. It doesn't give you a cause. For that you need to find the point where the run stopped being recoverable, and everything after that point is consequence rather than cause.

I do it backwards, which is counterintuitive but much faster than reading forward:

1. Read the final state. Name the concrete defect. Not "it failed," but "it wrote parse_config to return None on a malformed file where the spec says raise."
2. Walk backwards until you find the last turn where the agent still had a correct model of the problem.
3. The turn immediately after that is the divergence. Quote it verbatim.
4. Classify what happened there. Premature convergence, context loss, insufficient grounding, whatever your labels are.
5. Ask the counterfactual: what information, tool, or constraint would have prevented this?

Step five is the one that turns a description into something actionable, and it's the step most reviews skip.

Reading forward, you spend most of your attention on turns that don't matter, because early turns in a failed run usually look fine. Reading backward, you find the seam in a couple of minutes.

Three answers to the counterfactual, and only one is interesting

When you ask what would have prevented the failure, you get one of three shapes. Naming which one you're giving is most of the value of the review.

"A clearer prompt would have prevented this." Task defect. The spec was ambiguous, or it omitted a constraint the grader enforces. Very common, and worth catching, because a task defect contaminates every run against that task.

"A better tool or error message would have prevented this." Harness defect. This one gets misattributed to the model constantly. If an agent runs the same failing command four times, the usual cause is that the failure is coming back as a raw stack trace with nothing actionable in it. Feed it "selector .btn-submit not found; page contains .btn-primary, .btn-cancel" instead and the retry behaviour changes immediately. That was never a reasoning failure. It was an information failure.

Context loss lands here too. If the agent honoured a constraint for eight turns and then quietly stopped, look for a compaction boundary before you conclude the model has an attention problem.

"Nothing available would have prevented this." Genuine capability gap. This is the finding people actually want, which is exactly why you should reach for it last. Earn it by ruling out the other two first.

I've come to treat the harness/model split as the single most useful distinction in this work. It's also the one that separates a review written by someone who has built agent infrastructure from one written by someone who has only used it.

The honest cost

Trajectory evaluation is expensive. Outcome evals run in CI for pennies. Reading a sixty-turn transcript properly takes a human twenty minutes, and an LLM judge doing it for you inherits every bias an LLM judge has, plus a very long context to lose things in.

So don't do it on everything. What has worked for me:

• Outcome evals on every run, always, automated.
• Trajectory review on all failures, because that's where the diagnostic value is concentrated.
• Trajectory review on a sample of passes, specifically to catch the low-reasoning/high-performance quadrant. If you never audit your passes, you will never find the ones that were gamed. This is not paranoia. Recent work from Cursor found that 63% of one frontier model's successful resolutions on SWE-bench Pro retrieved a known fix rather than deriving it, and that was only visible because somebody went and looked at what the runs actually did.

The sampling rate matters less than the fact that it isn't zero.

What this looks like written down

A review that's useful to someone who can't see the transcript needs a fixed shape. I use roughly this, and the fixed shape is the point, because a hundred reviews in the same format are comparable and a hundred freeform notes are not:

## Review: <task> / <harness + model> / <date>

**Outcome:** pass | partial | fail
**Test integrity:** clean | suspected | confirmed gaming

### Trajectory
| Dimension | Score | Note |
|---|---|---|
| Tool selection | | |
| Argument construction | | |
| Result utilisation | | |
| Error recovery | | |
| Plan coherence | | |
| Task completion | | |

### Divergence
Turn N. "<verbatim quote>"
Classification: <label>
Counterfactual: <what would have prevented it>
Attribution: task defect | harness defect | capability gap

### Assessment
Two or three sentences on what this run says about the system,
not about this task.

That last section is the one people leave out, and leaving it out is the difference between an evaluation and a bug report. A bug report says this run broke. An evaluation says what this run predicts about the next thousand.

The short version

The final diff tells you whether you got lucky. The transcript tells you whether you'll get lucky again.

If you're shipping agents to production, you already know that a system which works four times out of five isn't a system. Trajectory evaluation is how you find out which of the four were real.