Finding 1: Harness affects cost more than correctness.

Fable 5 can cost twice as much for a 1.1-point gain in success rate. On SWE-bench Lite:
- Claude Code: 97.8% accuracy, $1.33/rollout
- Pi: 96.7% accuracy, $0.67/rollout

While Claude Code reaches the highest success rate on the SWE-bench Lite frontier, Pi and Codex often achieve similar success rates at lower cost across the models we test.

So you may be paying a hidden “harness tax” if you pick your harness based only on the success rate… 💸

(2/n)

Key finding 2: A simple harness can be competitive.

Pi reaches the Pareto frontier on both open-source benchmarks with 🛠️ just four tools 🛠️: read, write, edit, and bash…

The effectiveness of Pi and Codex demonstrates opportunities for open-source harness research.

And harness choice appears to mainly change the slope of the cost–quality curves, ie. at a given budget which harness can reach the target quality at a lower price faster.

(3/n)

Where is the harness tax coming from? 🧐

Agents can take similar numbers of turns at substantially different costs.

For Fable 5 on SWE-bench Lite, Pi and Claude Code average 15.4 and 15.3 turns per attempt, yet Claude Code costs about twice as much for a 1.1-percentage-point increase in success rate.

So what is happening each turn? One possible reason can even be observed at the first model call: Claude Code’s mean initial context is over 10× Pi’s, with longer instructions and larger tool schemas ‼️

As models become more capable, agents may need less scaffolding. For everyday tasks, harness design should therefore prioritize cost efficiency and reliability.

(4/n)

Key finding 3: Models may perform better with other harnesses than with their own.

And yes... you read that correctly: GPT-5.6 Sol can achieve a higher success rate in Pi than in Codex, and Opus 4.8 can achieve a higher success rate in Codex than in Claude Code 👀

A question many researchers ask: can open-source harness research win if models are co-trained with their native harnesses? Our study shows that today’s models can generalize well and be compatible across different harnesses. 🙌

(5/n)

More details in the blog post.

📄 harnesstax.github.io

We as the community should evaluate and select the model AND the harness together.

(6/n)

More broadly, the need for different harnesses depends on the role of coding agents.

💻 For day-to-day tasks, coding agents serve as an interface to model intelligence: they manage context, access tools, and execute tasks. As models become more capable, some of today’s scaffolding may become unnecessary.

General-purpose coding agents should therefore prioritize cost efficiency and reliability, as many tasks may not require fancy add-on features that add a harness tax.

🧪 For harder problems at the boundary of a model’s capabilities, including scientific discovery, agents may still benefit from more structured harnesses.

Harness research can therefore help models push the boundaries of knowledge, unlocking the next phase of intelligence in the process.

(7/n)

The insight comes from two open-source benchmarks we evaluated on, results may differ on other workloads.

Stay tuned for our future work on extending such evaluation to more setup! 🙌

(8/n)

This was an amazing & fun collaboration with @Andy_ShuoYang, @NegarEmpr , @infwinston, @istoica05, @matei_zaharia

We thank @arena, @LaudeInstitute, @AmazonScience for sponsoring this research.

We also thank @tianyin_xu and @mertcemri for providing valuable feedback.

@BerkeleySky @berkeley_ai 🙌

(9/n)

you can also check out the blog at @arena research blog series: arena.ai/blog/coding-ag… 😃

(10/n)