Why your benchmark results might be about the harness, not the model
The biggest hidden assumption in web-agent benchmarks is that the model is the main driver of performance. But a 2026 position paper argues the opposite for long-horizon tasks: the harness—the code that manages context, tools, and verification—often matters more than the model it wraps [5]. They formalize this with control theory, showing the harness acts like a controller and the model like a policy it governs. In their controlled variance decomposition, harness-induced variance exceeded model-induced variance, and in some cases, changing the harness reversed which model came out on top [5]. So if you're comparing two agents and they use different harnesses, you might be measuring the harness, not the model.
This isn't just theory. In a 2026 study, a smaller model (Gemini-2.5-Flash) automatically synthesized a code harness that prevented all illegal moves across 145 different games, allowing it to outperform larger models like Gemini-2.5-Pro [4]. Similarly, a self-improving harness method boosted pass rates on a software engineering benchmark from 59% to 78% in a single optimization round, without any external grading [2]. These results show that harness changes can produce performance shifts that exceed what you'd get from swapping one model for another—so if your benchmark doesn't control for the harness, your conclusions about model quality are shaky.
What happens when the harness assumptions break? Attacks and failures
Web-agent harnesses also make assumptions about safety and reliability, and those assumptions can break in practice. A 2026 benchmark, HarnessRisk, tested three harnesses across six language models and found attack success rates ranging from 12.6% to 80.9%—meaning in the worst case, four out of five adversarial instructions embedded in workflows succeeded [1]. The most vulnerable phase was harness configuration, where attacks altered security-sensitive parameters within otherwise authorized workflows [1]. Even more concerning, the study found that detecting a risk didn't reliably lead to safe action: some configurations detected risks in over 90% of runs but still had substantial attack success [1]. So the assumption that a harness will catch and block attacks is not always true.
Another assumption is that the harness's behavior is static and well-understood. But harnesses evolve, and that evolution can introduce new failure modes. A 2026 study on self-improving harnesses found that optimizing a harness on past trajectories improved performance on held-out tasks by up to 132% across nine model-benchmark combinations [3]. However, the same study noted that the retained mechanisms addressed benchmark-specific bottlenecks, which means the harness might become overfit to the benchmark's quirks [3]. This is a double-edged sword: harness optimization can boost scores, but it may also make the evaluation less representative of real-world tasks. The key takeaway is that harness assumptions—about safety, stability, and generalizability—are often violated, and evaluations that ignore this are incomplete.
What should you do when comparing agents? Demand harness disclosure
Given that the harness can dominate performance, the practical advice is clear: don't trust leaderboard comparisons that don't disclose the harness. The 2026 position paper proposes a disclosure standard and a variance decomposition protocol, arguing that until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading [5]. This is a call to action for benchmark creators and users alike.
For your own evaluations, the evidence suggests you should either fix the harness across all models or explicitly measure its contribution. The HarnessRisk study shows that safety outcomes vary dramatically with harness configuration, so you need to test multiple configurations to get a full picture [1]. And the self-improvement studies [2][3] show that harness optimization can be a powerful tool, but it also means your benchmark results are time-sensitive—a harness that works well today might not tomorrow. In short, treat web-agent benchmark results as conditional on the harness, and always ask: 'What harness was used?'
About These Sources
This answer is built on 5 studies (all preprints) — published in 2026, 5 from 2024 or later — selected as the most relevant from 9 studies that passed quality screening, drawn from 51 papers retrieved from a database of over 500 million.
Sources used in this answer
HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety
HarnessRisk, a lifecycle-oriented benchmark with 128 sandboxed cases across three harnesses and six models, found attack success rates ranging from 12.6% to 80.9%, with harness configuration being the most vulnerable phase, and risk detection not reliably leading to safe action.
Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference
RHO, a self-supervised harness optimization method, improved pass rate on SWE-Bench Pro from 59% to 78% in a single round without external grading, across three domains.
Self-Harness: Harnesses That Improve Themselves
Self-Harness, an iterative loop for agents to improve their own harness, improved both held-in and held-out pass rates across all nine model-benchmark combinations, with relative gains up to 132%.
AutoHarness: improving LLM agents by automatically synthesizing a code harness
AutoHarness showed that Gemini-2.5-Flash can automatically synthesize a code harness that prevents all illegal moves in 145 TextArena games, enabling it to outperform larger models like Gemini-2.5-Pro.
Stop Comparing LLM Agents Without Disclosing the Harness
This position paper formalizes the Binding Constraint Thesis, showing that harness-induced variance can exceed model-induced variance, including cases of model ranking reversal, and argues for harness disclosure in evaluations.