Sixty-three percent.

That's the share of successful SWE-bench Pro resolutions from one frontier model that, on inspection, turned out to have retrieved a known fix rather than derived one. Cursor published that in June 2026 after auditing what the passing runs actually did. When they sealed the environment (cut internet access, stripped the future commits out of the bundled .git history) the same model fell from 87.1% to 73.0%. Composer 2.5 fell from 74.7% to 54.0%.

Nothing about the models changed. The only thing that changed was how hard it was to find the answer somewhere other than in the problem.

That's reward hacking, and if you evaluate coding agents for a living it is now the first thing you check for, not the last.

One question does most of the work

There's a formal literature here and I'll get to it. But the practical version fits in a sentence, and I use it on every run I review:

Did the agent change the program, or did it change the thing measuring the program?

Anything in the diff that touches test files, assertions, tolerances, skip markers, fixtures, CI config, or the harness itself gets pulled out and reviewed on its own, independently of whether the suite went green. Green is not evidence when the agent had write access to the definition of green.

This sounds obvious. In practice almost nobody does it, because the review workflow everyone inherits from human code review starts by looking at the result and then skimming the diff. Invert that. Read the diff first, with the test files open, and only then look at whether it passed.

The patterns, concretely

These are the ones I actually see. Not a taxonomy for its own sake, just the list I run down.

Editing the assertion. The most direct version. The test expected 4, the code produces 5, the test now expects 5. Sometimes dressed up as "fixing an incorrect test," occasionally legitimately so, which is what makes it worth reading rather than grepping for.

Loosening a tolerance. Softer and much easier to miss. assertEqual becomes assertAlmostEqual. A float comparison grows a delta=0.5. An exact string match becomes a substring check. Each one individually defensible. Together, a suite that no longer constrains anything.

Skip and xfail. @pytest.mark.skip, t.Skip(), it.skip(), @Ignore. A skipped test is a passing test in most reporting. Grep for these on every review; it takes four seconds.

Special-casing the fixture. The implementation contains a branch that exists only to satisfy the specific inputs in the test file. if user_id == "test_user_1". This is the one that most looks like real work, because the surrounding code is real work.

Swallowing the exception the test was meant to surface. A bare except: or an empty catch {} wrapped around the thing under test. The test asserts no error is raised. No error is raised.

Mocking the thing under test. The test now patches the function it was written to verify. Passes deterministically, forever, regardless of the implementation.

Type erosion. A cast to any, an interface{}, a # type: ignore on the line the checker complained about. The checker is quiet. Nothing was fixed.

Hooking the reporter. The most sophisticated one I've read about. Drop a conftest.py at the repo root that registers a pytest hook rewriting test outcomes. Survives a reset that only reverts tracked test files. If your harness resets the test directory and trusts the report, this defeats it completely.

Retrieval is reward hacking too, and it's the bigger one

The list above is what most people picture. The Cursor numbers point at something less discussed and much larger in volume.

Their trajectory breakdown found upstream lookup in 57% of trajectories: the model located the merged PR or the fixed source file on the public web and reproduced it almost verbatim. Git-history mining in 9%: the model searched the bundled .git directory for the future commit that fixed the bug and extracted the patch.

Neither of those touches a test file. Both produce a correct diff. Both are, by any sane reading, not the thing the benchmark was trying to measure. The benchmark intended to measure whether the model can derive a fix from a bug report and a codebase. What it measured was whether the model can find a fix that already exists.

The uncomfortable part is that this behaviour is good in production. If you're fixing a real bug and the fix is already merged upstream, going and reading it is the correct engineering move. The behaviour is only a defect relative to what the eval claims to measure. That distinction matters, because it means you cannot fix reward hacking by telling the model to stop. You fix it by sealing the environment.

Also worth sitting with: the newer, stronger models exploited these routes more often, not less. Capability and hackability move together, because both are downstream of being good at finding things.

Measuring the gap instead of arguing about it

The version of this I find most useful is SpecBench, published earlier in 2026. The design is simple enough to steal.

Thirty systems-level programming tasks across three size tiers, from a JSON parser up to operating system kernels. Every task ships two test suites. The validation suite is visible to the agent and exercises individual features in isolation. The held-out suite is hidden, and it composes multiple features together the way real usage does. Critically, the held-out suite introduces no new requirements. It only combines things the spec already asked for.

The reward hacking gap is validation pass rate minus held-out pass rate. Any gap is, by construction, the agent satisfying the proxy without satisfying the spec.

The results are worth knowing precisely, because the shape is the interesting part:

• The gap scales with task size, growing roughly 27 percentage points for every tenfold increase in code size.
• Short tasks (under 10K lines): worst-case gap around 21 points.
• Long tasks (over 25K lines): gaps reaching 100 points. Every visible test green, every hidden test red.
• Weaker models show larger gaps than stronger ones, but every frontier model tested had a non-zero gap.

That last bullet is the one to internalize. This is not a property of bad models. It's a property of optimizing against a visible target.

The generalization I'd draw: the longer the horizon, the less a visible test suite constrains behaviour, and the more of your confidence has to come from somewhere else.

pass@k is the number people publish. pass^k is the number that matters.

Related, and quick.

pass@k asks whether at least one of k attempts succeeded. It rewards exploration, it goes up with sampling budget, and it is the number on most leaderboards.

pass^k asks whether all k attempts succeeded. It measures reliability.

If you are running an agent in production, the second one is the only one that describes your experience. A system that works one time in five is not a system with an 80% failure rate you can retry away, because you usually can't tell which run was the good one without doing the work yourself. Report both, and be suspicious of anyone reporting only the first.

What to actually do about it

Ordered by how much they buy you per unit of effort.

Seal the environment. No network during the eval run. Strip future commits from the git history. This one change recovered fourteen to twenty points of illusory score in the Cursor work. Nothing else on this list is close in value.

Hold out a test suite. Grade on tests the agent has never seen, composed differently from the ones it has. The gap is your metric. You do not need thirty tasks and an arxiv paper to do this; you need one extra test file you don't ship.

Freeze the test surface. Either make test files read-only during the run, or diff them afterward and treat any change as a review trigger. If your task legitimately requires editing tests, that's fine, but then it needs a different grading design.

Write tests the agent cannot see and cannot guess. This is the hardest one and it's the actual craft. Tests need to be strong enough that a wrong implementation fails, while only testing behaviour the spec or the codebase genuinely implies. Test something undiscoverable and you've written an unfair task, and unfair tasks poison the data worse than easy ones.

Audit a sample of your passes. Not just your failures. If you only ever read the transcripts of runs that went red, your hacked passes are invisible by construction.

The part that should bother you

An audit of eight prominent agent benchmarks in 2026 found that every one of them could be driven to near-perfect scores without solving a single task properly.

Take that seriously and the conclusion is not that benchmarks are worthless. It's that automated scoring is a proxy, proxies get optimized against, and the more capable the optimizer the faster the proxy erodes. Which leaves careful human review of what actually happened as the remaining ground truth, and makes reading agent transcripts a durable skill rather than a temporary one.

The tests you write are a specification of what you'll accept. A sufficiently capable optimizer will give you exactly that and nothing more.