Harness Handbook: Solving the "Where to Edit" Problem in AI Agent Evolution

TL;DR

As AI agents move from simple scripts to production-grade "harnesses," finding where to make changes becomes harder than writing the code itself. Harness Handbook introduces a behavior-centric map that connects what an agent does (e.g., "confirming task completion") to exactly where that logic lives in the codebase. It improves edit-plan quality by up to 18.9% while actually saving on LLM token costs.

Concept: The Behavioral Gap in Codebases

When a developer wants to change an agent's behavior—say, requiring three confirmation prompts instead of one—they think in terms of behavioral stages. However, code repositories are organized by files and modules.

In production harnesses like Codex or OpenHands, a single behavior might be scattered across state managers, API wrappers, and prompt templates. Existing tools like RAG or repository maps help you search for keywords, but they don't tell you how those pieces orchestrate a specific outcome. This is the "Behavior Localization" bottleneck.

Methodology: Putting Behavior First

The researchers developed a pipeline to automatically generate a "Handbook" from raw source code.

1. The Three-Level Hierarchy (L1-L3)

Instead of a file tree, the Handbook provides:

• L1 (System Overview): High-level architecture and execution model.
• L2 (Component Overview): Responsibilities and state transitions of specific execution stages.
• L3 (Unit Deep Dive): Direct links to the functions or file regions that implement that behavior.

2. The Construction Pipeline

The framework extracts static facts (call graphs, signatures) and uses an LLM to "cluster" these into execution stages.

• Function-as-leaf: Used for smaller, high-density harnesses (e.g., Terminus-2).
• File-as-leaf: Used for repository-scale systems (e.g., Codex).

Behavior-Guided Progressive Disclosure (BGPD)

Instead of feeding an entire codebase into a long-context window, BGPD mimics how a senior lead developer would guide a junior:

1. Stage Selection: Use the Handbook's L1/L2 to find relevant execution stages.
2. State Tracing: Follow "State Registers" to see which other stages read or write relevant data.
3. Source Verification: Only after identifying potential sites does the agent open the actual source files to verify the logic.

Experimental Results: Precision & Efficiency

The authors tested the framework against "Search-Hostile" requests—changes that are difficult to find using standard keyword searches.

Key Findings:

• Higher Accuracy: Handbook-assisted agents achieved significantly better F1 scores in identifying all required edit sites.
• Lower Cost: By navigating a structured index rather than brute-force searching code, planner token usage dropped by ~10%.
• Model Level-up: With the Handbook, a mid-tier model like DeepSeek-V4-Pro performed localization task as well as (or better than) "frontier" models like GPT-5.5 or Opus 4.8.

Critical Insights: Beyond Just Code Generation

The most profound takeaway is that for AI agents to self-evolve, they need behavioral memory. Standard code-repair agents often suffer from "scope bloat"—changing things they shouldn't—because they lack a mental model of the system's intent.

By keeping a "Handbook" synchronized with the repository, we provide agents with an operational behavioral representation. It serves not just as a map for humans, but as a structural substrate for agents to perform "self-auditing" and "impact analysis" before they ever touch a line of code.

Conclusion and Future Work

The Harness Handbook proves that the future of automated software engineering isn't just better code generation, but better repository understanding. The next logical step? Self-evolving harnesses where the agent updates its own handbook as it modifies its own behavior, creating a closed-loop system of autonomous improvement.

For more details, check out the project at ruhan-wang.github.io/Harness-Handbook/