Self-Harness: The Rise of the Self-Reshaping Agent
Self-Harness: Harnesses That Improve Themselves
This paper introduces Self-Harness, an iterative framework that enables LLM-based agents to autonomously refine their own "harness" (prompts, tools, and runtime logic). By implementing it on Terminal-Bench-2.0, the authors achieved significant performance gains across diverse models like MiniMax M2.5, Qwen3.5, and GLM-5, with held-out pass rates increasing by up to 21.4% absolute points.
TL;DR
In the evolution of AI agents, we have transitioned from simple prompt engineering to complex "harnesses"—the scaffolding of tools, system prompts, and runtime rules that dictate how an LLM interacts with the world. However, these harnesses are usually handcrafted by humans. Self-Harness flips the script, allowing a fixed LLM to analyze its own execution failures and rewrite its own operating rules. The result? Performance boosts of up to 138% on complex terminal tasks, tailored specifically to the "personality" of the underlying model.
Problem: The "One-Size-Fits-All" Harness Trap
Currently, when we build an agent, we provide it with a harness (e.g., ReAct, Claude Code). But a harness that works for GPT-4 might lead Qwen or GLM into an endless loop. Different models have different tool-use habits, sensitivities to instruction, and failure modes.
Historically, solving this meant:
- Human Engineering: Slow, expensive, and unscalable.
- Meta-Harnessing: Using a 1-trillion parameter model to "fix" a smaller model. This is expensive and often misses the local nuances of the smaller model's behavior.
The authors argue that for a system to truly mature, it must be capable of self-creation—identifying its own weaknesses and patching its own interface.
Methodology: The Propose-Evaluate-Accept Loop
The core of Self-Harness is a three-stage iterative loop that treats harness improvement as an empirical state transition rather than a shot in the dark.
1. Weakness Mining
Instead of looking at failures as isolated errors, the system clusters them by "Verifier-Grounded Failure Signatures." It asks: Did we fail because of a timeout? A missing file? A syntax error? By grouping these, the agent sees patterns (e.g., "I always forget to create the output file until the last second").
2. Harness Proposal
The same model that failed the task is now asked to act as a "Developer." Provided with summarized evidence of its failures, it proposes minimal code changes to its own configuration—such as adding a new system instruction, changing a tool's retry logic, or enabling a runtime control policy.
3. Proposal Validation (The Regression Gate)
Crucially, No edit is accepted blindly. Every proposal must pass a Regression Test. It is tested on "Held-in" tasks (the ones it failed) and "Held-out" tasks (new challenges). An edit is only merged if it improves performance on one without breaking the other.
Experiments: Model-Specific Evolution
The researchers tested Self-Harness on Terminal-Bench-2.0 using three distinct model families: MiniMax M2.5, Qwen3.5, and GLM-5.
Model Personality Discoveries:
- MiniMax M2.5: The system discovered that this model explored too much and forgot to write output files. Self-Harness added a "create artifact early" instruction and a "max tool message" limit to force progress.
- Qwen3.5: This model tended to retry failed commands endlessly. Self-Harness implemented a specialized middleware that triggered recovery prompts after tool errors.
- GLM-5: It struggled with environment persistence. The evolved harness added rules to ensure shell settings persisted across sessions.
Critical Analysis: Why This Matters
The beauty of Self-Harness lies in its Inductive Bias. It assumes that the model isn't broken, but the interaction is. By keeping model weights fixed, it provides a "non-parametric" way to improve AI.
Limitations:
- Benchmark Specificity: There is a risk that the harness overfits the specific types of tasks in Terminal-Bench.
- High Compute Cost: Running multiple evaluation loops and regression tests is expensive in terms of inference tokens.
- Bootstrap Requirement: You still need a "minimal" base harness to start; the agent cannot yet build itself from absolute zero.
Conclusion
Self-Harness moves us closer to "Autonomous Agent Engineering." Instead of human experts painstakingly tuning prompts for every new model version, we can let the models "mature" through their own experience. As the authors quote Henri Bergson: "To exist is to change... to mature is to go on creating oneself endlessly."
