[Tsinghua | HIT] Natural-Language Agent Harnesses: Turning Agent Scaffolds into Executable Research Artifacts

TL;DR

A new paradigm from Tsinghua and HIT researchers proposes Natural-Language Agent Harnesses (NLAHs). Instead of burying agent logic (like ReAct loops or memory handling) in brittle Python code, NLAHs turn this logic into portable, editable natural language. Supported by an Intelligent Harness Runtime (IHR), this approach allows researchers to treat agent orchestration as a modular, scientific object, resulting in a significant performance jump on the OSWorld benchmark (from 30.4 to 47.2).

The "Harness" Problem: Why Your Agent Actually Fails

In the current AI landscape, we often credit (or blame) the base model (e.g., GPT-4, Claude 3.5) for an agent's success. However, the "harness"—the messy glue code that handles multi-step reasoning, tool retries, and state management—is often the real hero.

The problem? This harness logic is usually scattered across framework defaults, hidden prompts, and tool adapters. This makes it impossible to perform a fair "apples-to-apples" comparison between different agent systems. Are you winning because of your model, or because your Python script happens to have a better retry logic?

Methodology: Lifting Code into Language

The researchers define the Pattern Layer—the high-level strategy—as something that should be externalized. They introduce two core components:

1. NLAH (The Artifact): A structured text file defining roles (Solver, Verifier), stage structures (Plan → Execute → Verify), and "Contracts" (what must be produced to move to the next step).
2. IHR (The Runtime): An LLM-powered engine that reads the NLAH and the current environment state to decide the next orchestration move.

Figure 1: The IHR executes NLAHs by interpreting roles, contracts, and state conventions over task instances.

The Secret Sauce: File-Backed State

One of the most profound insights of the paper is the File-Backed State module. Long-horizon agents often fail because their memory is "ephemeral" (lost in the context window). By forcing the agent to write its state to a canonical workspace (TASK.md, state/, artifacts/), the system ensures that even if a context window is truncated, the "spine" of the task remains addressable and robust.

Experimental Insights: Structure is a Double-Edged Sword

The researchers tested NLAHs on SWE-bench Verified (coding) and OSWorld (computer use).

Key Result 1: Behavioral Realignment

The "Full IHR" configuration significantly changes how an agent spends its budget. For example, in the "TRAE" harness, 90% of tokens were spent by delegated child agents rather than the parent "coordinator." This shows the harness isn't just a prompt; it's a genuine management system.

Key Result 2: The "Over-Engineering" Trap

Interestingly, adding more structure (like a "Verifier" stage) didn't always improve scores. In some cases, a complex verifier might judge a patch as "solved" based on its own internal logic, even if it fails the benchmark's literal evaluator.

Table 1: Ablation studies showing how different modules (Self-Evolution, File-Backed State) affect performance across benchmarks.

Code-to-Text Migration: A Surprising Gain

In a "paired migration" study, the team converted the existing OS-Symphony code harness into an NLAH. Performance jumped from 30.4 to 47.2.

Why? The NLAH realization re-centered the agent's behavior around durable artifacts rather than "brittle GUI repair." Instead of getting stuck in a loop of clicking the wrong button, the language-guided agent was more likely to switch to the shell or edit a package directly to get a "completion certificate."

Critical Analysis & Conclusion

This work signals a shift from "Prompt Engineering" to "Harness Engineering."

Takeaways:

• Portability: We can now "share" agent strategies as NLAH files, regardless of the underlying software stack.
• Scientific Rigor: By externalizing logic, we can finally ablate "Self-Evolution" or "Multi-Candidate Search" as independent modules.
• Caveat: Natural language is less precise than code. The authors admit that some proprietary service-side behaviors simply cannot be captured in text yet.

The future of AI agents belongs to those who build the best scaffolds. With NLAHs, those scaffolds have finally become a readable, executable, and optimizable search space.