How reliable are AI agents on long-running tasks?
The short answer: not very. The most realistic and rigorous study available — ScienceBoard, published in 2025 — tested state-of-the-art AI agents (GPT-4o, Claude 3.7, UI-TARS) on 169 real scientific workflows across biochemistry, astronomy, and geoinformatics. Agents had to use professional software, navigate dynamic interfaces, and complete multi-step tasks. The overall success rate was just 15% [2]. That means for every 100 long-running tasks you hand to an AI agent, roughly 85 will fail or produce incorrect results. This is not a niche finding: the same thesis that spans AI agent research from 2020 to 2024 concludes that LLMs are "inconsistent in complex procedural tasks" and struggle with tasks requiring dynamic interaction [1].
Even on simpler, well-defined freelance programming tasks — where the work is structured and testable — the best model (Claude 3.5 Haiku) earned only about $1.52 million out of a possible total, meaning it left a substantial portion of tasks unsolved or incorrect [3]. The authors explicitly note a "gap between performance on structured tasks versus the true complexity of real-world freelance jobs" [3]. So the 15% figure from ScienceBoard is likely the more honest picture for messy, long-running tasks.
Why do agents fail, and can we fix them?
Agents fail for three main reasons. First, they lack robust state management: if a task is interrupted or requires remembering context across many steps, current agents lose track. A 2026 framework called Agentic State Management (ASM) proposes a formal state machine to let agents "suspend and resume their work seamlessly after interruptions," but this is a proposed solution, not a proven one [5]. Second, agents suffer from unpredictable failures like hallucinations, execution errors, and inconsistent reasoning. A 2026 self-healing framework that detects these failures and triggers adaptive replanning showed "significantly increased task success rates" in early tests, but again, this has not been validated at the scale of the ScienceBoard benchmark [6]. Third, agents struggle to adapt to unfamiliar tasks that weren't in their training data. One approach is to feed them external knowledge (e.g., manuals) at runtime, which improved accuracy but still fell short of reliable performance [1].
A 2025 evaluation framework paper argues that current benchmarks overemphasize accuracy while ignoring cost, robustness, and long-term adaptability — the very dimensions that matter for long-running tasks [7]. So the field is aware of the gaps, and new frameworks are emerging, but none have yet closed the reliability chasm.
Can agents ever be trusted without human oversight?
Not yet, and perhaps not soon. A 2021 paper on human-agent teamwork argues that for complex, real-world settings, adjustable autonomy — where the human can step in at any point — is essential [8]. This is not a weakness of AI; it's a recognition that long-running tasks in dynamic environments require judgment calls that current agents cannot reliably make. A separate 2021 paper on robo-advisors in finance emphasizes that trust in automated systems depends on explainability (XAI) and that humans perceive robots as complements, not substitutes [9]. The implication: even if agents improve, the most trustworthy setup for long-running tasks will likely involve human oversight for the foreseeable future.
The most optimistic path forward comes from a 2026 dissertation that proposes building "foundation agents" by combining internet-scale data, large language models for reasoning, and unified vision-language-action architectures [4]. The author acknowledges this is an "open challenge" and that current systems face "three key limitations": scarcity of interactive training data, difficulty grounding high-level goals into long-horizon behavior, and fragmentation of perception, language, and motor control [4]. So even the most ambitious roadmaps admit we are not there yet.
About These Sources
This answer is built on 9 studies (1 peer-reviewed, 8 preprints) — published from 2021 to 2026, 7 from 2024 or later, collectively cited 123 times — selected as the most relevant from 10 studies that passed quality screening, drawn from 43 papers retrieved from a database of over 500 million.
Sources used in this answer
Solving Real-World Tasks with AI Agents
A 2024 thesis spanning 2020-2024 research concludes LLMs are inconsistent in complex procedural tasks requiring dynamic interaction, and proposes new benchmarks, a Python-program formalization for agents, and external-knowledge augmentation to improve accuracy.
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
A 2025 benchmark (ScienceBoard) of 169 real scientific workflows found that state-of-the-art agents (GPT-4o, Claude 3.7, UI-TARS) achieved only a 15% overall success rate, showing they cannot yet reliably assist scientists in complex workflows.
Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task Success at Scale
A 2025 benchmark on freelance programming tasks (simulated from real job postings) found the best model (Claude 3.5 Haiku) earned $1.52M of a possible total, but the authors note a gap between structured tasks and real-world complexity.
Building Foundation Agents with Internet Knowledge and Large Language Models
A 2026 dissertation proposes building foundation agents by combining internet-scale data, LLMs for reasoning, and unified vision-language-action architectures, but acknowledges three key limitations: scarce interactive data, difficulty with long-horizon goals, and fragmented system components.
The Agentic State Management (ASM) Framework for Long-Running AI Tasks
A 2026 paper introduces the Agentic State Management (ASM) framework, a formal state machine and data model enabling agents to suspend and resume long-running tasks after interruptions, but it is a proposed framework without large-scale validation.
A Self-Healing Framework for Reliable LLM-Based Autonomous Agents
A 2026 paper proposes a self-healing framework for LLM agents that integrates failure detection, reliability assessment, and automated recovery via adaptive replanning; early experiments showed significantly increased task success rates.
Beyond Accuracy: A Multi-faceted Evaluation Framework for Real-World AI Agents
A 2025 paper argues that current AI agent evaluations overemphasize accuracy while ignoring cost-effectiveness, robustness to dynamic environments, and long-term adaptability, and proposes a multi-faceted evaluation framework.
Human-Agent Teamwork and Adjustable Autonomy in Practice
A 2021 paper on human-agent teamwork advocates for adjustable autonomy in complex real-world settings, arguing that human oversight remains essential for reliable performance in dynamic environments.
Robo-Advising: Less AI and More XAI?
A 2021 paper on robo-advisors in finance emphasizes that trust in automated systems depends on explainability (XAI) and that humans perceive robots as complements, not substitutes, for decision-making.