Self-Harness: When LLM Agents Become Their Own System Architects

TL;DR

In the evolution of AI agents, we’ve moved from "Prompt Engineering" to "Harness Engineering"—the complex scaffolding of tools, memory, and runtime rules. Until now, these harnesses were handcrafted by humans. Self-Harness flips the script, allowing a fixed LLM to diagnose its own failures and rewrite its own operating code. The result? Dramatic performance gains (up to 138% relative improvement) without touching a single model weight or hiring a single human engineer.

The Motivation: The "One Size Fits All" Fallacy

Modern LLM agents are defined by their harness: the system prompt, the available tools, and the failure-recovery logic. However, a harness designed for GPT-4 might be disastrous for Qwen or GLM. Since different models have distinct "personalities," tool-use habits, and hallucination patterns, they require model-specific environments.

Manually tuning these for every new model release is a bottleneck. Previous attempts at automation (Meta-Harness) used "Teacher" models to fix "Student" models. Self-Harness removes the teacher, proving that even a "weak" model contains enough self-awareness to patch its own execution logic when shown evidence of its own failures.

Methodology: The Self-Correction Loop

The researchers break down Self-Harness into a rigorous three-stage iterative process:

1. Weakness Mining (The Diagnostic)

Instead of looking at failures as random noise, the system clusters execution traces by failure signatures.

• Physical Intuition: If a model consistently fails because it forgets to save a file, the system groups these "missing artifact" errors together, creating a clear signal that the harness needs a "save-early" rule.

2. Harness Proposal (The Architect)

The system invokes the same model to act as a proposer. Guided by the clustered failures, it suggests minimal edits to its own harness code (e.g., adding a verification step or changing a tool's instruction).

3. Proposal Validation (The Gatekeeper)

Every "bright idea" from the proposer must survive a Regression Test. A candidate edit is only adopted if it improves performance on "held-in" tasks without breaking "held-out" (unseen) tasks. This prevents the model from simply overfitting to a single error.

Experimental Results: Model-Specific Evolution

The beauty of Self-Harness is that it discovers different solutions for different models:

• MiniMax M2.5: The system learned to enforce early artifact creation.
• Qwen3.5: The system implemented dependency pre-checks and retry disciplines.
• GLM-5: It focused on environment persistence across shell sessions.

On Terminal-Bench-2.0, the gains were universal. Most notably, Qwen3.5 saw its pass rate jump from 15.1% to 36.0%. These aren't just marginal tweaks; they are fundamental shifts in how the agent approaches tasks.

Case Study: Solving the "Endless Loop"

In one specific instance (MiniMax M2.5), the initial harness allowed the agent to explore a dataset indefinitely until it timed out. Self-Harness identified this "stalled loop" pattern and automatically injected a runtime control policy that limited tool messages, forcing the agent to stop exploring and start writing the answer.

Strategic Insight & Conclusion

Self-Harness suggests a future where agent developers don't write complex prompts. Instead, they provide a minimal seed harness and an evaluation suite, allowing the agent to "grow" its own optimal operating environment through experience.

Limitations: Currently, the system relies on high-quality verifiers (pass/fail signals). In open-ended creative tasks where "success" is subjective, the validation gate becomes much harder to build.

Final Takeaway: This work moves us closer to "Recurrsive Self-Improvement." By treating the harness as a modular, optimizable software layer, we enable agents to mature not by changing their "brains" (weights), but by refining their "workspace" (harness).