Here is the run that made us write this up.

One of the hardest tasks, the pass rate (all model efforts) on DeepSWE is 38%, Pi and Claude Code both fixed it.

Pi took 90 turns and $2.50. Claude Code took 381 turns and $64.36. Roughly 26x more for the same fix.

If you just want to know what to use:

Codex if you don't want to think about it. Best pass rate, medium cost.
Pi if the same job runs a thousand times and the bill adds up.
Exo if retries are cheap and you'd rather it quit early than grind.
DSH if you care about wall-clock and don't mind playing with knobs.

Something we learned:

With implicit prefix caching, running a task once during debugging leaves it warm for hours. Test a harness on Tuesday, benchmark it Wednesday, and it shows up cheaper than it should. Nothing in the logs tells you why.

So no benchmark task was touched before the formal run, and every one of the 360 trials is a fresh restore of one checkpoint, memory state included.

v1.0 focused on software engineering and terminal tasks. Next we will test the full harness × model grid. Much of what we observed points to harness-model fit rather than harness quality, and we want to identify which combinations maximize pass rates while minimizing cost.

Leaderboard and the long version: frontierharness.org/