How often do these agents fail, and what kinds of mistakes do they make?
The failure rates are alarmingly high, especially in safety-critical contexts. In a large evaluation of 21 different LLMs on 500 physician-validated medical vignettes, models chose a deliberately sabotaged ("sham") guideline over the correct one 40.6% of the time—meaning they got the answer wrong in nearly 2 out of every 5 decisions [1]. The worst failures occurred on the most dangerous changes: models missed removed warnings, deleted allergy information, and ignored contraindication violations with failure rates ranging from 54.2% to 61.7% [1]. This means that in over half of cases involving critical safety information, the AI picked the harmful option.
A separate viewpoint article reviewing current medical AI agents confirms that these problems are not isolated. It notes that citation fidelity is inconsistent across models, with subtle misinterpretations and unreliable references still common, and that safety constraints become less predictable within multi-step research pipelines [3]. Together, these findings show that the agents are not just making minor errors—they are failing on the very tasks that matter most for patient safety.
What causes these failures—is it a fundamental flaw or a fixable bug?
The failures stem from deep, structural weaknesses in how these agents process information, not just from simple bugs. The clinical guideline study found that the models were heavily influenced by presentation bias: they chose the first option presented 72.7% of the time, regardless of whether it was correct or sabotaged [1]. This shifted accuracy from 82.3% down to 36.7% depending solely on which option appeared first—meaning the AI's decision was largely a coin flip based on order, not on understanding.
Another study on LLM-based robotic systems points out that current agents lack transparency in their retrieval and evidence-ranking mechanisms, raising concerns about reproducibility and hidden biases [4]. The viewpoint article adds that the retrieval processes remain opaque, making it hard to audit why an agent chose a particular source [3]. These are not superficial issues—they indicate that the agents lack robust reasoning and are easily swayed by superficial features of the input.
Can these agents be useful in any real-world setting right now?
Yes, but only as assistive tools under close human supervision, not as autonomous decision-makers. The same studies that highlight failures also show genuine strengths. For example, the Agent Laboratory framework, which uses LLMs to conduct literature reviews, run experiments, and write reports, was able to produce machine learning code that achieved state-of-the-art performance, and it reduced research costs by 84% compared to previous autonomous methods [2]. However, human involvement at each stage significantly improved the quality of the final output [2].
The viewpoint article explicitly argues that these agents should be embraced as "assistive research tools rather than pseudoexperts," valuable for accelerating information gathering but not for replacing rigorous human judgment [3]. A study on LLM-based NPCs in virtual reality showed that agents can produce context-relevant responses with appropriate facial expressions and gestures, but this was in a controlled, low-stakes entertainment setting [5]. The consensus across all five papers is clear: these agents can boost efficiency and reduce costs, but they are not reliable enough to operate without human oversight, especially in medicine, finance, or any domain where errors have serious consequences.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2024 to 2026, 5 from 2024 or later, 1 in Q1 journals, collectively cited 79 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 57 papers retrieved from a database of over 500 million.
Sources used in this answer
When Agentic LLMs Trust Poisoned Tools: Vulnerability of Clinical LLMs to Adversarial Guidelines.
In a large evaluation of 21 LLMs on 500 clinical vignettes, models chose a sabotaged medical guideline 40.6% of the time, with failure rates up to 61.7% for safety-critical changes, and showed strong presentation bias (72.7% chose the first option).
Agent Laboratory: Using LLM Agents as Research Assistants
The Agent Laboratory framework, using LLMs for literature review, experimentation, and report writing, achieved state-of-the-art code performance and reduced research costs by 84%, but human feedback at each stage significantly improved output quality.
Deep research agents: Major Breakthrough or Incremental Progress for Medical AI? (Preprint)
A viewpoint article argues that deep research agents represent incremental progress, not a paradigm shift, citing inconsistent citation fidelity, opaque retrieval processes, and a risk of amplifying errors at scale; it recommends using them as assistive tools rather than autonomous experts.
Agentic LLM-based robotic systems for real-world applications: a review on their agenticness and ethics
A survey of LLM-based robotic systems validated in real-world applications introduces an 'agenticness' classification and an ethical evaluation framework, highlighting gaps in transparency, safety guardrails, and human oversight that must be addressed before deployment.
Building LLM-based AI Agents in Social Virtual Reality
An LLM-based AI agent deployed as an NPC in VRChat using GPT-4 demonstrated the ability to produce context-relevant responses with appropriate facial expressions and gestures, but the evaluation was preliminary and limited to a low-stakes entertainment setting.