Can harness-environment co-evolution improve agentic abstention reliability?

Abstention is not the same as task success

The paper targets a reliability gap it calls agentic abstention: an agent keeps acting when a request is underspecified or the environment lacks the prerequisites for success, instead of stopping. The authors argue that continuing to act in such cases can degrade task performance, add API cost and latency, and even produce unintended or irreversible actions.

To measure this, HERA-BENCH uses matched pairs. Each pair contains a solvable instance and a should-abstain instance that share the same public task instruction and tool interface; only a controlled change to the environment decides whether the request can be completed. For N matched pairs, Abstain is the percentage of should-abstain instances where the agent correctly abstains, Act is the percentage of solvable instances it completes, and Pair requires correct behavior on both instances of the same pair. Over-abstain counts explicit abstention on a solvable task and is reported as a diagnostic, not as the complement of Act, because incorrect answers and unsuccessful actions can occur without abstention.

This distinction matters because a harness that simply refuses more often can raise Abstain while lowering Act. The paper's central comparison is therefore whether abstention gains come at the cost of feasible-task completion, and it reports both metrics alongside Pair.

Controlled environment mutations create verifiable infeasible tasks

HERA's first stage builds executable environment-task ecosystems from scratch in three steps: environment construction, task generation, and environment mutation. Starting from a collection of environment themes, a deep-research agent with a fixed research policy and a set of research tools builds a topic-aligned structured database that is iteratively expanded and refined.

The mutation step is what produces abstention cases. Rather than writing infeasible tasks directly, the pipeline applies controlled environment mutations that transform solvable tasks into cases requiring abstention, so each feasible-infeasible pair shares the same task request while the environment mutation determines whether the request can be completed.

Verification is part of the construction, not an afterthought. Authors review each generated pair for clear requests, plausible scenarios, and intended feasibility; for the feasible version they run a reference solution and check it meets all requirements, and for the infeasible version they identify what prevents completion and check whether another allowed approach could still satisfy the request. The paper states explicitly that a model's failure alone does not establish that a task is infeasible, and that the two versions must share the same user goal with the feasibility difference explained by the specified environment change.

Failures drive both harness changes and new environments

The second stage is the co-evolution loop. Using the construction procedure from stage one, HERA alternates environment evolution and new task construction with harness adaptation. In each round, paired execution trajectories are analyzed to identify two kinds of failure: failing to complete feasible tasks, and failing to abstain on infeasible tasks. Those failures guide targeted harness changes and generate new execution environments and tasks aimed at the weaknesses just exposed.

The motivation is diminishing signal. The authors argue that optimizing a harness against a fixed task set can only expose a limited set of failures, and that as the harness adapts, the same examples provide less information about where it still fails. Continued improvement therefore requires the task distribution itself to adapt.

The training pool reflects this schedule: it starts with 20 task pairs and expands to 50 through six additions of five pairs, with a fixed validation set of 15 pairs used for checkpoint selection. HERA-BENCH contains 60 pairs and serves as the common benchmark for comparing selected harnesses across models.

Co-evolution leads on abstention without losing feasible tasks

On held-out evaluation tasks, the evolved harness improves abstention accuracy from 61.7% to 83.3% and feasible-task completion from 68.3% to 76.7%, the highest Abstain and Act among the compared methods. The comparison set separates two families. Static harness baselines are H0 at 61.7% Abstain and 68.3% Act, and H0 plus an abstention prompt at 68.3% Abstain and 63.3% Act. Harness optimization baselines are harness-only evolution at 76.7% and 71.7%, task-augmented harness evolution at 80.0% and 75.0%, and task generation from H0 failures at 78.3% and 75.0%.

The abstention-prompt row is the clearest illustration of the trade-off the paper is trying to avoid: explicit prompting raises Abstain but lowers Act. Harness evolution improves both metrics relative to H0, and co-evolution reaches the highest values on both, which the authors present as evidence that environmental evolution adds benefit without loss on feasible tasks.

The advantage also shows up on later task batches. Co-evolution and task generation from H0 failures score identically in the first rounds T0 and T1 but diverge afterwards; on T6, co-evolution reaches 86.0% Abstain and 88.0% Act against 76.0% and 82.0% for generation from initial failures. The authors read this as support for updating task generation with current harness feedback rather than only with the failures of the initial harness.

Newly evolved batches start low, then the harness recovers

The co-evolution dynamics are tracked per fixed task batch. B0 is the initial set of 20 task pairs, and each later batch contains five new task pairs introduced after a round of environment evolution; the batches do not overlap, and Ti is the cumulative training pool through round i.

Newly generated batches tend to begin at lower performance than earlier batches under the same harness, which the authors interpret as environment evolution successfully targeting current weaknesses. The concrete example is H5, which performs substantially worse on the newly introduced B6 than on B0 and most previous batches; after B6 is incorporated into harness evolution, the update to H6 raises both Abstain and Act on B6 from 60.0% to 80.0%.

Feasible-task performance is reported as stable across harness evolution. Act on earlier batches shows some fluctuation but a slight overall positive trend, and the paper notes that the evolution procedure does not explicitly target Act when generating new tasks. The claim is bounded to the observed batches: successive harness updates did not degrade feasible-task performance in this setup.

One evolved harness transfers across 19 other LLMs

The final harness learned with HERA on the GPT-5.6-Luna base model is fixed and evaluated on 19 other closed- and open-weight LLMs without any model-specific optimization. Across those models, abstention accuracy improves by 15.3 percentage points on average, with per-model gains ranging from 6.7 to 40.0 percentage points in Abstain and from 1.7 to 28.3 percentage points in Pair.

The gains are not confined to weak baselines. GPT-6-Astra improves from 81.7% to 91.7% in Abstain and from 70.0% to 80.0% in Pair, even though it already outperformed GPT-5.6-Luna under H0. The paper's own framing is that harness improvements developed through co-evolution can transfer across the evaluated model families, while the magnitude of the gains varies by model.

Over-abstention is reported separately for the same cross-model evaluation. Most models show 0.0% over-abstention under both H0 and the final harness, with small nonzero values in a few cases, such as GPT-5.6-Luna at 1.7% and Mistral Large 3 at 15.0% under the final harness. This is a diagnostic on solvable instances, not a substitute for the Abstain or Pair numbers.

The cost trade-off and where the evidence stops

Applying HERA is not free within a model. Across 14 models on HERA-BENCH, it increases API cost and token usage by median factors of 2.09 and 3.31 respectively. The paper's cost argument is comparative rather than absolute: despite that per-model overhead, some models reach or approximate the best base-harness performance at much lower cost. GPT-5.6-Luna with HERA achieves 70.0% paired success, matching GPT-6-Astra with the base harness, at an estimated average API cost of $0.0091 versus $0.0610 per episode, roughly 85% lower.

The scope conditions are set by the benchmark design. Evaluation uses HERA-BENCH with 60 pairs, a fixed validation set of 15 pairs, and a training pool that grows from 20 to 50 pairs; the cross-model transfer result covers the 19 evaluated models and the specific harness learned on GPT-5.6-Luna. The paper also notes that task-level disjointness does not imply disjoint environment families or source domains, which require separate grouping and overlap checks.

The practical priority this paper supports is not abstention alone. The decision-relevant comparison is the joint one: a harness is preferable when it raises Abstain without lowering Act, and the cost comparison only becomes favorable when a cheaper model with the evolved harness matches a stronger model's paired success. On this benchmark, co-evolution is the configuration that satisfies both conditions, and the transfer result extends that comparison across model families rather than establishing it for every deployment setting.

Sources used in this answer