Why task completion rates alone hide critical failures
A single number like '70% success' tells you nothing about why the other 30% failed, or whether the 'successes' introduced hidden problems. A 2025 study that evaluated three popular open-source agent frameworks on 34 programmable tasks found a task completion rate of about 50%, but the real value came from their systematic failure analysis [3]. They developed a three-tier taxonomy of failure causes: planning errors (the agent chose the wrong strategy), task execution issues (it couldn't carry out the steps), and incorrect response generation (it produced a wrong output even after executing). Without this breakdown, you'd just see a mediocre score and have no idea where to improve. The same study proposed actionable fixes for planning and self-diagnosis, showing that failure analysis directly leads to better agents.
Another 2025 study on autonomous web agents used as testers found that while their best agent (PinATA) achieved a promising 94% specificity (correctly identifying passing tests), it only got about 60% of verdicts correct overall [1]. The qualitative evaluation identified specific limitations—like fragility when the application structure changed—that a simple pass/fail rate would mask. The authors concluded that to build resilient agents, you need to understand these failure modes, not just count successes.
Privacy and trust: metrics that matter as much as success
An agent that completes a task but leaks your personal data is not a success—it's a liability. A 2025 benchmark called AgentDAM specifically measures whether web agents follow the privacy principle of 'data minimization': using sensitive information only when necessary [4]. When they tested agents built on GPT-4, Llama-3, and Claude, all three were prone to using unnecessary sensitive information during tasks. This is a failure that a task completion rate would never catch. The study also proposed a prompting-based defense that reduced leakage, showing that privacy evaluation can drive concrete improvements.
Similarly, a 2022 study on a medical AI agent (BreastScreening-AI) evaluated how clinicians interacted with the system, not just whether it made correct diagnoses [2]. They measured acceptance, satisfaction, and time-to-diagnose. The Clinician-AI scenario reduced false positives by 27% and false negatives by 4%, and 91% of clinicians reported positive expectations and satisfaction. The time-to-diagnose dropped by 3 minutes per patient. These human-centered metrics—trust, workflow impact, error reduction—are essential for real-world deployment and go far beyond a task completion rate.
Robustness and adaptability: how agents handle the unexpected
A brittle agent that succeeds on familiar tasks but fails on slight variations is not truly capable. The 2025 study on autonomous web agents as testers found that their agents struggled when application structure changed, a classic robustness failure [1]. They called for developing agents that are 'resilient and reliable' in the face of such changes. This is a dimension that task completion rates on a fixed benchmark cannot capture.
Another 2025 study introduced WebPilot, a multi-agent system that uses Monte Carlo Tree Search (MCTS) to handle the uncertainty and complexity of real web tasks [5]. On the WebArena benchmark, WebPilot achieved a 93% relative increase in success rate over a previous tree-search method, but the key innovation was strategic exploration—the ability to adapt plans based on new observations, not just follow a rigid script. The authors explicitly contrast this with agents that rely on 'expert-designed policies specific to certain states and actions,' which lack flexibility. Evaluating adaptability requires testing on unseen tasks or with dynamic environments, not just measuring completion on a static set.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2022 to 2025, 4 from 2024 or later, 1 in Q1 journals, collectively cited 118 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 35 papers retrieved from a database of over 500 million.
Sources used in this answer
Are Autonomous Web Agents Good Testers?
In a benchmark of 113 manual test cases on three web apps, the best autonomous test agent (PinATA) achieved about 60% correct verdicts and 94% specificity, but qualitative analysis revealed fragility to application changes, showing that task completion alone misses robustness issues.
BreastScreening-AI: Evaluating medical intelligent agents for human-AI interactions
In a study with 45 clinicians from nine institutions, the Clinician-AI scenario reduced false positives by 27% and false negatives by 4%, while 91% of clinicians reported positive satisfaction and time-to-diagnose dropped by 3 minutes per patient—showing human interaction metrics are critical beyond accuracy.
Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
Evaluating three open-source agent frameworks on 34 tasks found a ~50% completion rate, but failure analysis revealed a three-tier taxonomy (planning errors, execution issues, incorrect response generation), providing actionable improvements that a single success rate would hide.
AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents
The AgentDAM benchmark tested GPT-4, Llama-3, and Claude agents for privacy leakage and found all prone to using unnecessary sensitive information, highlighting that privacy evaluation is essential beyond task completion.
WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration
WebPilot, a multi-agent system using Monte Carlo Tree Search, achieved a 93% relative increase in success rate on WebArena over prior methods, demonstrating that strategic exploration and adaptability are key evaluation dimensions beyond raw completion.