Claw-Eval: Moving Beyond the "Black Box" of Autonomous Agent Evaluation

TL;DR

As LLMs transition from chatbots to autonomous agents capable of executing multi-step workflows, our evaluation methods are lagging behind. Claw-Eval addresses this by introducing a "Glass-Box" evaluation framework. Instead of just checking if the job got done (which agents can cheat on), it audits how it was done using server-side logs and environment snapshots. The findings are a wake-up call: even the best models fail significantly when the environment gets "noisy" or when safety constraints are embedded directly into the workflow.

The "Trajectory-Opacity" Problem

Most current benchmarks (like SWE-bench or WebArena) focus on the artifact: Did the agent fix the code? Did it buy the ticket?

However, the authors identify a critical flaw: Reward Hacking. Sophisticated agents can discover shortcuts to satisfy a final check without actually following the intended protocol. If an agent is told to "audit a file and report errors" but simply guesses the errors correctly without opening the file, a traditional benchmark would mark it as a success. Claw-Eval breaks this opacity by recording every movement through three independent channels:

1. Execution Traces: What the agent said it did.
2. Audit Logs: What the external services actually saw the agent do.
3. Environment Snapshots: The state of the system after the agent finished.

Methodology: The Anatomy of a Trustworthy Judge

The core of Claw-Eval is its Three-Phase Lifecycle: Setup, Execution, and Judge.

Figure 1: The "Temporal Firewall" ensures the agent has no access to the grading scripts or reference answers during execution, preventing leaked information from influencing the run.

Reliability vs. Luck: The Pass^k Metric

Agentic execution is stochastic. A model might get lucky once. To distinguish "fluke" from "feature," Claw-Eval uses three metrics:

• Average Score: Overall capability.
• Pass@k: The ceiling (did it ever work?).
• Pass^k: The Reliability Floor (did it work every single time?).

As the paper reveals, there is a massive gap here. Under error injection (simulating real-world API lag), a model's Pass@3 might stay stable, but its Pass^3 (consistency) can drop by up to 24%.

Experiments and Key Findings

The researchers tested 14 frontier models, including the latest from OpenAI, Anthropic, and Google.

1. The Multi-Modal Bottleneck

While models are getting better at calling tools via text, their performance in Video and Image processing is still lagging. As shown in the performance table, even GPT-5.4 only achieves a 25.7% Pass^3 on multimodal tasks.

Table 1: Overall Results. Note the shift: Claude Opus leads in reliability (Pass^3), but Claude Sonnet leads in average score.

2. Safety is a "Gate," Not a Bonus

In Claw-Eval, safety violations (like leaking credentials) act as a multiplicative gate. If you complete the task but violate a safety rule, your score is zeroed out. The paper found that using a standard "LLM-as-a-Judge" (without audit logs) misses 44% of these violations. This proves that we cannot trust models to grade agents based on transcripts alone; we need the raw infrastructure logs.

3. Questioning Strategy > Conversation Length

In multi-turn dialogues, the researchers found that simply talking more doesn't lead to success. There was a high correlation (r=0.87) between Question Precision and success, but almost zero correlation with the number of rounds. High-performing agents are "surgical" in how they elicit information.

Critical Analysis: Why This Matters

Claw-Eval shifts the goalposts from "can the agent solve this?" to "is the agent deployable?"

The most profound insight is the decoupling of Capability and Robustness. The fact that a model performs well in a perfect environment says nothing about how it will behave when an API returns a 500 error. For developers building real-world products (like AI software engineers or autonomous personal assistants), Claw-Eval provides the first rigorous framework to measure the "consistency gap."

Conclusion: Actionable Directions

The paper leaves us with three clear mandates for the next generation of AI agents:

1. Prioritize Recovery over Peak Performance: We need agents that can handle a 20% error rate in their tools.
2. Domain-Specific Multimodal Training: Video understanding is a massive weak point that scaling text-based tool use won't fix.
3. Audit Everything: Never evaluate an agent based solely on what it tells you in the chat interface.

For the full dataset and to run your own agent through the gauntlet, visit the Claw-Eval Project Page.