Harness-of-Harness: Making Coding Agents Improve Across Loops Instead of Across Prompts

TL;DR

The paper studies autonomous software development: an LLM-based coding agent should receive a high-level specification and produce a complete, runnable, usable artifact without step-by-step human guidance. Its answer is Harness-of-Harness (HoH), a wrapper that repeatedly invokes the same harness-model configuration as a Project Planner, Developer, and QA Tester while carrying both code and evaluation evidence across loops. On three benchmarks and three harness-model pairs, HoH@3 improves over the Vanilla baseline, with the Abstract reporting an average relative gain of 52.25 percent and a maximum gain of 82.86 percent. In a separate multi-day case, 70 HoH loops turn a product requirements document into a first-person shooter game with a closed issue backlog of 65 resolved items out of 81 recorded issues.

Background positioning

This is not a model-training paper and not an AutoHarness-style attempt to optimize the harness code itself. It sits at the orchestration layer above existing harnesses such as Codex CLI, OpenCode, and Pi Coding Agent, treating the harness-model pair as fixed while changing how development passes are coordinated. The claim is therefore narrower but more practical: a software project can be advanced over many loops if every loop inherits both a writable artifact and a structured evidence packet, instead of inheriting only the latest files. The contribution is primarily systems-level, and its value depends on whether the loop protocol can be reproduced across models, tasks, and tool stacks.

Problem and motivation

The paper’s motivation is precise: autonomous greenfield software development is not merely a longer execution trajectory, but a state management problem. In a conventional harness, the model sees a local session and writes code, but the reasons for prior changes, the failures observed during QA, and the behaviors that should now be preserved can disappear once the context window ends. Section 3.1 describes three concrete failure mechanisms: earlier requirements and design decisions can be forgotten; locally reasonable fixes can violate constraints elsewhere; and incomplete artifacts can be accepted as complete because functional or quality gaps are not detected. This is the actual failure mode being attacked, not just low accuracy on a single test.

The author intuition is also concrete: software engineering already has mechanisms for bounded increments, independent acceptance, and change history, but agentic coding systems often collapse these mechanisms into one model session. HoH therefore borrows iterative and incremental development as a control structure, but implements it as agent contracts rather than a fixed workflow. The paper does not offer a formal theorem that the loop converges; the justification is conceptual plus empirical ablation, which is appropriate for a system paper but leaves some design choices dependent on the chosen benchmarks.

Core mechanism

1. The bottleneck is not raw execution length, but a one-shot mapping

The paper begins by abstracting end-to-end software development as:

where $M$ is the frozen language model, $H$ is the coding harness through which the model interacts with the environment, $\mathcal{S}$ is the public software specification, and $A$ is the final artifact. In the paper’s argument, this mapping is the failure point because it hides the intermediate decision process: the model is asked to choose, implement, verify, preserve, and extend behavior all inside one trajectory. If the task is small and self-contained, the one-shot formulation may work. The boundary becomes visible when $\mathcal{S}$ describes an integrated system, such as a playable game with mechanics, assets, UI, audio, and mission logic, because no single pass can simultaneously decide the project scope, write to the artifact, and independently accept the resulting behavior. The real problem is that the final $A$ contains no durable record of which sub-claims were verified, which gaps remain, and which constraints later changes must respect.

2. HoH separates artifact continuity from evidence-guided planning

The framework then replaces the one-shot mapping with a loop transition:

where $A_t$ is the software artifact after loop $t$, including source, configuration, resources, and metadata, and ${\mathcal{E}}_t$ is the execution evidence state obtained by evaluating that artifact against the specification and the current development objective. This formula is the conceptual heart of HoH. The Planner consumes $\mathcal{S}$ and ${\mathcal{E}}_{t-1}$ to produce a bounded development document, the Developer consumes that document plus $A_{t-1}$ and writes the new artifact, and the Tester evaluates a frozen read-only candidate to produce ${\mathcal{E}}_t$. The key design choice is not that three agents appear, but that the state passed forward is split: code carries implementation, while evidence carries validated knowledge. Without $A_t$ continuity, each loop would have to reconstruct the project from code alone, which Section 3.3 says can overlook unmet requirements, repeat known work, or regress validated behavior. Without ${\mathcal{E}}_t$ continuity, every new plan would be driven only by the specification, losing failures and preservation constraints discovered during earlier testing.

The role boundaries are also important. Appendix A.2 makes them explicit: the Planner cannot modify production code, the Developer is the single writer to the active workspace, and the Tester inspects and executes but does not silently repair. This is a deliberate tradeoff. If the Tester could edit the artifact, it would blur acceptance with implementation and make it unclear whether a verified behavior came from the candidate under evaluation or from later repairs. If the Planner could write code, it would need implementation-level technical autonomy and local project access, which would dilute the object-selection role. The paper’s runtime instead enforces read/write permissions and structured output schemas; it constrains what can be delivered, not how the agent reasons inside that envelope. That distinction matters because it avoids brittle handcrafted workflows while still making each step auditable.

3. Evidence becomes a claim-evidence record rather than an unstructured report

The evidence state is not simply a text summary. Section A.4 defines it as normalized records derived from the specification and the current development document:

Here ${\mathcal{E}}_t^{\mathrm{ver}}$ contains records whose status is verified, and ${\mathcal{E}}_t^{\mathrm{gap}}$ contains records whose status is a gap. A verified record can become a preservation constraint in the next plan; a gap record becomes an update target and a validation requirement. The formula also clarifies what does not happen: missing evidence, unmet requirements, regressions, and failures are not silently treated as success, because they are placed in the gap partition rather than inferred as complete. This is the paper’s operational answer to the premature-completion problem. It does not prove that QA Tester judgments are correct, but it turns acceptance into a schema-bound decision that can be passed between loops. If either partition were removed, the system degenerates into a weaker design: without ${\mathcal{E}}^{\mathrm{ver}}$, validated behavior loses its preservation role; without ${\mathcal{E}}^{\mathrm{gap}}$, the Planner has no structured signal about what remains unfinished.

Experimental evidence

Main benchmark gains

Table 1 reports the headline benchmark comparison across three harness-model configurations. The main experiments use $T=3$, meaning three planning-development-testing iterations, while Vanilla performs one standard development pass under the same harness-model configuration and benchmark-provided initial state.

These numbers support three claims. First, the improvement is not tied to a single configuration or task family: every harness improves on GameCraft-Bench, FrontierSWE, and ProgramBench after three loops. Second, the gains are not merely from starting with a stronger model, because OpenCode with DeepSeek-V4-Pro begins at the lowest Vanilla scores yet reaches 48.98 on GameCraft-Bench and 44 percent dominance on FrontierSWE. Third, the magnitude can be substantial: Codex adds 21.93 GameCraft points and 27 percentage points of FrontierSWE dominance; Pi adds 16.85 ProgramBench pass-rate points. The main table does not report generation replicates, so the comparison should be read as task-average performance under the described protocol rather than as a statistical significance statement.

Figure 5 then extends the FrontierSWE analysis beyond three loops for the Codex configuration. It shows that dominance can continue improving when evaluated against an 11-checkpoint comparison pool consisting of Vanilla and HoH@1 through HoH@10.

In that figure, Vanilla obtains 27.33 percent dominance, HoH@3 is 39.33 percent, HoH@9 reaches the best value of 76.00 percent, and HoH@10 is 72.67 percent. The improvement is suggestive for long-horizon development, but the comparison pool differs from Table 1, so 71 percent and 72.67 percent are not directly comparable across the two analyses. Also, the paper reports the best checkpoint over ten loops, which is useful for diagnosis but would need a held-out stopping rule if used as a final artifact policy.

The gains are not only more tokens

A critical threat to interpretation is that HoH simply spends more compute by running multiple passes. Table 2 isolates this by comparing HoH against Vanilla Continuation, which uses the same model and harness but receives an additional prompt to continue development without a separate planning and QA structure.

The pass-controlled table is one of the strongest parts of the paper. HoH@1 already beats one-pass Vanilla while using only 2.88M tokens versus 2.59M, and HoH@2 with 5.67M tokens outperforms three-pass Vanilla Continuation with 6.33M tokens. This means the gain cannot be fully explained by a larger inference budget alone; the structure of each pass adds value. Still, HoH@3 consumes 8.41M tokens per task on average, more than triple Vanilla, so cost-effectiveness remains a design tradeoff. Section B.7 also notes that token accounting differs across providers and is used for within-configuration comparisons, not cross-provider cost claims.

Cross-iteration mechanisms are not decorative

Table 3 ablates the three major cross-loop mechanisms on GameCraft-Bench with Codex and GPT-5.5 high.

The pattern maps directly onto the conceptual design. Freezing the development document after the first plan lowers the score by 8.13 points, showing that the Planner cannot simply issue one initial spec and then let later iterations guess what to change. Removing evidence feedback lowers the score by 6.28 points even though artifact warm-start remains, indicating that planning without test results loses the ability to prioritize failures and preserve validated behavior. Removing warm-start lowers the score by 7.85 points and increases mean tokens from 8.41M to 11.12M, because each iteration rebuilds from the empty workspace despite having evidence-conditioned plans. The ablation is not perfect, since every variant still uses a single run per task-condition, but it gives unusually clean support for the framework’s state channels.

Figure 4 provides a further component-level decomposition on GameCraft-Bench: HoH@3 improves all four rubric dimensions for all three configurations, and for Codex the Functional Visuals component rises from 48.67 to 74.23 while Art and Presentation rises from 45.28 to 65.28. That result matters because the benchmark is not only about code running, but about playable, understandable, polished products. It also shows that independent QA is not merely a gate for compilation; it is shaping content depth and presentation quality over successive loops.

Multi-day case study

The Fusepoint case studies a different question: whether a fixed HoH configuration can sustain coherent project evolution when implementation constraints, validated behavior, and observed failures accumulate over many loops. The system ran HoH with Codex CLI and GPT-5.6-Sola high reasoning effort for 70 development loops, starting from an empty workspace containing only a product requirements document for a single-player narrative first-person shooter. Human involvement was limited to restoring network or API availability, not planning, implementation, debugging, testing, or acceptance.

By Loop 70, 65 of 81 recorded issues were closed and 16 remained unresolved; 17 issues were reopened after an earlier closure because a subsequent change caused previously verified behavior to fail again. Section 5.2 interprets this as evidence of two forms of continuity: the versioned workspace lets implementation work accumulate, while the issue history and evidence packets keep unfinished work, verified behavior, and regressions visible to later planning. This is a much weaker form of evidence than benchmark aggregate scores, but it is qualitatively valuable because it demonstrates the intended behavior in an open-ended, long-running setting rather than only on fixed tasks.

Depth insights and summary

The central contribution is not “use three agents” in a generic sense, but the separation between artifact state, evidence state, and role authority. A normal coding agent often has one local model session that tries to plan, write, and judge the same artifact. HoH breaks those decisions into distinct invocations with explicit permissions and structured outputs, and then makes the later loop depend on both the implementation and the validated claims from the previous evaluation. That is what enables repair, extension, and preservation to be reprioritized as the artifact evolves. The strongest evidence for this reading is not only the headline benchmark gains, but the pass-controlled comparison and the ablation showing that plan update, evidence feedback, and warm-start each contribute.

The limitations are specific and testable. Appendix B.7 states that every reported task-condition score is obtained from one valid run, with failed attempts replaced when infrastructure errors occur, so the paper does not average across multiple generations of the same task-condition pair. The selected clients also do not expose a common reproducible generation seed, and temperature and top-p are not overridden, which means model nondeterminism is present but not measured across seeds. FrontierSWE evaluates only 15 of 17 tasks because two tasks were unavailable or environment-incompatible, and the multi-day case uses additional domain-specific tools, skills, and external assets beyond the benchmark protocol. These caveats do not invalidate the main trend, but they narrow the scope: HoH is shown to help on the tested benchmark and case settings, not to be a universal guarantee for autonomous software engineering.

The paper’s future work should focus on calibrating the evidence channel. If QA Tester marks behavior as verified when the replay trace, screenshot, or white-box record only weakly supports it, the next Planner may treat broken functionality as a preservation constraint; the issue-reopen data from Fusepoint hints that this is a real risk. The next practical step is to track false-positive and false-negative rates in ${\mathcal{E}}_t^{\mathrm{ver}}$ and ${\mathcal{E}}_t^{\mathrm{gap}}$, perhaps by using hidden benchmark tests or human audits to compare normalized claims against ground-truth behavior. Another step is adaptive termination: HoH@3 often improves over HoH@1 and HoH@2, but the Pi ProgramBench score peaks at HoH@2 in Table 1, and the 10-loop FrontierSWE figure selects a best checkpoint. A deployment-oriented follow-up would need an online criterion for when to stop, when to accept regression repair, and when a plan update has become redundant.