Why do harness tests overpromise on real-world performance?
Harness tests often measure an agent's ceiling on a curated set of tasks, but real-world tasks are messier and more varied. A 2026 study on harness adaptation found that small language models (SLMs) could match large language models (LLMs) on 7 of 21 task-model pairs, but only after the harness was automatically optimized for those specific tasks [1]. This means that a harness that works well in a benchmark may fail in production if the tasks are less repetitive or the model's base capabilities are weaker—the same study noted that adaptation works best for tasks with more repetitive workflows [1].
The gap between best-case and typical-case is also about cost. The study showed that an optimized SLM agent could recover 89.7% of LLM performance at 4% of the cost [1]. But that's the best-case scenario after tuning; without that tuning, SLMs typically fall short when dropped into a harness designed for a frontier LLM [1]. So a harness test that doesn't account for cost or model-harness fit can make a solution look viable when it isn't.
What specific aspects do harness tests often overlook?
Harness tests can miss the need for reproducibility and structured observation. A 2026 paper introducing VERO argues that agent optimization—iteratively improving an agent through edit-execute-evaluate cycles—requires capturing both intermediate reasoning and downstream execution outcomes [2]. Standard harnesses often only record final outputs, missing the reasoning traces that are crucial for debugging and improving agents [2]. Without these traces, a harness test may show a pass/fail but not reveal why an agent failed or how to fix it.
Another overlooked aspect is the definition of the harness itself. A 2026 conceptual analysis points out that the term 'agent harness' is used loosely, sometimes meaning the whole product (like Claude Code), sometimes the evaluation scaffold (like SWE-bench), and sometimes an SDK or plugin [3]. This ambiguity means that two different 'harness tests' might be evaluating entirely different things, making results incomparable [3]. The paper proposes a constitutive definition with necessary and sufficient conditions, but until that's widely adopted, harness tests may be measuring different constructs under the same name.
When are harness tests most likely to give a false sense of security?
Harness tests are most misleading when they ignore the stochastic nature of LLM agents. Unlike deterministic code, agents interleave code with stochastic LLM completions, so a single run may not be representative [2]. VERO addresses this by using budget-controlled evaluation and versioned agent snapshots to ensure reproducibility [2]. Without such controls, a harness test might pass on one run and fail on another, and the test wouldn't reveal that variability.
Harness tests also mislead when they don't account for the cost-performance trade-off. The 2026 study found that SLMs can be 90% cheaper but only if the harness is adapted to their strengths [1]. A harness test that only reports accuracy, without cost, can lead to deploying an expensive LLM when a cheaper SLM with an adapted harness would suffice—or vice versa, deploying an SLM that fails because the harness wasn't adapted [1].
About These Sources
This answer is built on 3 studies (1 peer-reviewed, 2 preprints) — published in 2026, 3 from 2024 or later — selected as the most relevant from 3 studies that passed quality screening, drawn from 48 papers retrieved from a database of over 500 million.
Sources used in this answer
Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation
In a study across seven business tasks and three SLM families, optimized harnesses improved performance on 16 of 21 task-SLM pairs, with seven pairs closing the SLM-LLM gap and the best SLM recovering 89.7% of LLM performance at 4% of the cost, but adaptation worked best only for repetitive tasks and sufficiently capable SLMs.
VeRO: An Evaluation Harness for Agents to Optimize Agents
The VERO evaluation harness provides versioned agent snapshots, budget-controlled evaluation, and structured execution traces to enable reproducible agent optimization, highlighting that standard harnesses often miss intermediate reasoning and execution outcomes.
What makes a harness a harness: necessary and sufficient conditions for an agent harness
A conceptual analysis proposes a constitutive definition of an agent harness with necessary and sufficient conditions, noting that the term is used loosely across products, evaluation scaffolds, and frameworks, which can lead to inconsistent evaluation.