[ArXiv 2026] ClawBench: Why Your AI Agent Fails at Booking a Doctor's Appointment
ClawBench: Can AI Agents Complete Everyday Online Tasks?
ClawBench is a novel evaluation framework featuring 153 "write-heavy" everyday online tasks (e.g., booking appointments, submitting job applications) across 144 live production websites. It introduces a safe interception mechanism and an Agentic Evaluator to measure how frontier AI agents handle real-world web complexity beyond static sandboxes, revealing that even Claude Sonnet 4.6 achieves only a 33.3% success rate.
TL;DR
While modern AI agents boast high scores on static benchmarks, they fall apart when faced with the "messy" reality of the live web. ClawBench is a new, high-stakes benchmark of 153 everyday tasks on 144 real-world websites. It proves a sobering reality: the best current model (Claude 4.6) fails 2 out of every 3 simple real-world tasks.
Key Position: This work shifts the focus of agent evaluation from "sandbox toy problems" to "live-production reliability," introducing a safe way to test agents on sites like Expedia or LinkedIn without actually committing irreversible transactions.
The Reality Gap: Sandboxes vs. The Wild
Digital agents are currently in a "Local Maximum." In controlled environments like WebArena, they look like geniuses. But these sandboxes are often static, simplified, and lack the "friction" of the real internet—pop-ups, slow-loading JavaScript, and complex anti-bot measures.
The authors of ClawBench argue that we haven't been testing the most important thing: Write-heavy operations. Most current benchmarks only ask agents to "find information" (Read). ClawBench asks them to "Get things done" (Write), such as submitting a job application or ordering groceries.
Figure 1: Traditional benchmarks vs. ClawBench. Note the sharp performance drop on the right.
Methodology: Interception & Five-Layer Traceability
How do you let an AI try to "Buy a Laptop" on Amazon for testing purposes without actually spending $2,000?
1. The Interception Layer
ClawBench uses a Chrome Extension and a Chrome DevTools Protocol (CDP) server. Humans manually annotate the "Final Submission Request" for every task. When the agent clicks "Place Order," the system intercepts the outgoing HTTP request, logs the payload for evaluation, and blocks it from reaching the server. This allows for Ecological Validity (the site behaves exactly as it would for a human) with Zero Side Effects.
2. Five-Layer Recording
To move beyond a simple "Pass/Fail" result, ClawBench records:
- Full Video via virtual display.
- Step-by-step Screenshots.
- Full HTTP Traffic (including headers/body).
- Agent Internal Reasoning (the 'thought' traces).
- Low-level Browser Actions (exact coordinates of clicks).
3. The Agentic Evaluator
Instead of fragile "string matching" on the final page, ClawBench uses a Claude Code sub-agent. This evaluator compares the agent's five-layer trace against a human ground-truth trace. It checks if the agent filled the right fields with the right info, rather than just checking if it reached the "Thank You" page.
Experimental Results: A Reality Check
The results (Table 2 in the paper) are a "wake-up call" for the industry.
| Model | Overall Success Rate (SR) |
|---|---|
| Claude Sonnet 4.6 | 33.3% |
| GLM-5 | 24.2% |
| GPT-5.4 | 6.5% |
| Kimi K2.5 | 0.7% |
The most shocking finding is the performance of GPT-5.4, which plummeted to 6.5% on ClawBench despite being a top-tier model. This suggests that some models might be over-optimized for existing benchmarks (Overfitting) but lack the robust visual and procedural grounding needed for the ever-changing live web.
Table 2: Breakdown of success rates across different life categories.
Critical Analysis & Insights
Why do they fail? The failure analysis reveals that current agents struggle with "Multi-step Multi-platform" logic. For instance, an agent might successfully fill out a form but fail to notice a "Verify Email" pop-up or a specific field requirement that wasn't in the initial prompt.
Limitations:
- Temporal Decay: Because it uses the "Real Web," websites change. The benchmark requires constant maintenance (which the authors address by open-sourcing the pipeline).
- Authentication: High-stakes tasks (like banking) still require pre-set session cookies or manual logins, which remains a hurdle for fully autonomous evaluation.
Conclusion: The Road to Reliability
ClawBench proves that "General Intelligence" in agents is still far off. We have the "Reasoning," but we lack the "Reliability." By providing a framework that is both safe and realistic, ClawBench sets the new standard for what a production-ready AI agent must conquer.
The industry value is clear: stop optimizing for static HTML and start building agents that can survive the chaos of the live internet.