How easily can LLM research agents be tricked by bad information?
Very easily, according to the largest study in this set. Researchers tested 21 different LLMs on 500 medical vignettes, asking each model to choose between an authentic clinical guideline and a sham version with one adversarial change—like a removed warning, a deleted allergy, or a wrong dose [1]. Across 10,500 decisions, the models picked the dangerous sham 40.6% of the time. For the most safety-critical changes—like removing allergy information or introducing dosing errors—failure rates climbed to 54.2% to 61.7% [1]. That means in over half of those cases, the LLM chose a guideline that could directly harm a patient.
The study also uncovered a simple but powerful vulnerability: presentation bias. The models favored whichever option appeared first in the prompt 72.7% of the time [1]. When the sham was placed first, accuracy dropped to 36.7%; when the real guideline was first, accuracy jumped to 82.3% [1]. This means the order in which information is presented—something a malicious actor could easily control—can flip the model's decision from safe to dangerous. The authors warn this risk is especially acute in low-resource settings where AI agents might serve as primary health gatekeepers [1].
Beyond poisoning, what other risks do these agents carry?
Even when not deliberately attacked, LLM research agents have serious limitations that are easy to overlook. A review of deep research agents in medicine found that while they can rapidly gather and structure information, their citation fidelity is inconsistent—subtle misinterpretations and unreliable references are common [2]. Their retrieval and evidence-ranking processes are opaque, making it hard to spot hidden biases or reproduce results [2]. The authors also warn that overreliance on AI-generated syntheses risks eroding clinicians' critical appraisal skills and introducing automation bias, where users trust the output simply because it looks comprehensive [2].
A separate survey on the security and privacy of LLM agents catalogues a broader set of threats: these agents handle vast amounts of data and interact with humans and other systems, creating new attack surfaces for data leakage, manipulation, and adversarial control [3]. The survey notes that these vulnerabilities are only beginning to be studied systematically [3]. Together, these findings converge on a clear message: the risks are not hypothetical—they are documented, measurable, and span from deliberate poisoning to subtle, everyday failures in accuracy and transparency.
Can human involvement fix these problems?
Yes, but only if it is built into the process, not added as an afterthought. A study of Agent Laboratory—an autonomous LLM framework that conducts literature reviews, experiments, and report writing—found that human feedback at each stage significantly improved the quality of the final research output [4]. The authors emphasize that the system is best used as an assistive tool to accelerate information gathering, not as a replacement for human judgment [4]. This aligns with the recommendation from the medical review: these agents should be embraced as research assistants, not pseudo-experts, and their use requires transparent retrieval architectures, robust benchmarking, and explicit educational integration to preserve human evaluative reasoning [2].
The evidence is clear that without human oversight, the risks of error and manipulation are high. But with careful design—including independent verification, ranking safeguards, and human-in-the-loop feedback—these tools can be useful. The key is not to underestimate the risks, but to design systems that acknowledge and mitigate them from the start.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2026, 4 from 2024 or later, 2 in Q1 journals, collectively cited 161 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 53 papers retrieved from a database of over 500 million.
Sources used in this answer
When Agentic LLMs Trust Poisoned Tools: Vulnerability of Clinical LLMs to Adversarial Guidelines.
In a study of 21 LLMs on 10,500 agentic decisions across 500 medical vignettes, models chose a sham guideline with one adversarial change 40.6% of the time, with failure rates over 60% for safety-critical edits; presentation bias (favoring the first option 72.7% of the time) was the dominant factor.
Deep research agents: Major Breakthrough or Incremental Progress for Medical AI? (Preprint)
A review of deep research agents in medicine argues they represent incremental progress, not a paradigm shift, citing inconsistent citation fidelity, opaque retrieval processes, risk of automation bias, and lack of real-world deployment evidence.
The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
A comprehensive survey of LLM agent security and privacy categorizes threats including data leakage, manipulation, and adversarial control, noting these vulnerabilities are only beginning to be systematically studied.
Agent Laboratory: Using LLM Agents as Research Assistants
Agent Laboratory, an autonomous LLM research framework, reduced research costs by 84% compared to prior methods, but human feedback at each stage significantly improved output quality, supporting an assistive rather than replacement role.
Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization
A dynamic LLM agent collaboration framework (DyLAN) improved accuracy by up to 25% on MMLU subjects by selecting optimal agent teams, demonstrating that agent composition and structure matter for task performance.