How bad are false positives with AI coding agents?
The short answer: very bad, if you use a general-purpose AI chatbot as your only security scanner. In a 2023 study that tested ChatGPT against 11 traditional static code analysis tools on 92 real-world PHP vulnerabilities, ChatGPT detected 62-68% of the bugs—roughly double the best traditional tool's 32% detection rate [1]. That sounds great, but the catch is that ChatGPT also had a 91% false positive rate, meaning 91 out of every 100 alerts it raised were incorrect. Even the worst traditional tool had a lower false positive rate (82%) [1]. So if a team relied solely on ChatGPT, they'd be flooded with mostly useless alerts, which is exactly the kind of overwhelm the question asks about.
The study's authors concluded that combining ChatGPT with traditional static analyzers could get the best of both worlds—higher detection with fewer false alarms—since the two approaches catch different types of bugs [1]. This suggests that AI alone isn't the answer, but AI plus traditional tools might be.
Can newer AI agents reduce false positives?
Yes—and this is where the evidence gets more encouraging. A 2026 study introduced a multi-agent LLM (large language model) system called QASecClaw that specifically targets false positive reduction in static application security testing (SAST) [2]. Instead of just flagging everything, it uses a Filter Agent that reviews the source code and classifies each finding as a true or false positive. While the abstract doesn't give exact numbers, the approach is designed to cut the noise that overwhelmed teams in the earlier ChatGPT study.
Similarly, a 2024 study used an LLM (Llama) as an AI agent in a Security Operations Center (SOC) to automatically identify false positive alerts from network sensors [3]. The system works by extracting key details from each alert (like IP addresses and filenames), enriching them with threat intelligence databases, and then running them through predefined investigation playbooks designed by security experts. The result is automated triage that reduces the manual burden on human operators [3]. Both studies show that when AI agents are purpose-built for false positive reduction—rather than used as general-purpose chatbots—they can help, not hurt.
What should a team actually do?
The evidence points to a clear strategy: don't replace your existing security tools with an AI agent—augment them. The 2023 study found that combining ChatGPT with traditional static analyzers would catch more vulnerabilities than either alone, while keeping false positives in check [1]. The 2024 and 2026 studies show that specialized AI agents designed for false positive reduction can further cut the noise [2][3]. And the 2023 user study warns that teams need to guard against over-reliance on AI [5].
In practical terms, this means: (1) keep your existing static analysis tools as a baseline, (2) add an AI agent that is specifically trained or configured to filter false positives (not a general chatbot), (3) use human-designed playbooks to validate AI decisions, and (4) invest in developer security training so that AI assistance doesn't breed overconfidence. The research consistently shows that AI can improve security, but only when it's part of a thoughtful, layered system—not a magic bullet.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2026, 3 from 2024 or later, collectively cited 180 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 42 papers retrieved from a database of over 500 million.
Sources used in this answer
New Tricks to Old Codes: Can AI Chatbots Replace Static Code Analysis Tools?
In a 2023 study testing ChatGPT against 11 traditional static code analyzers on 92 PHP vulnerabilities, ChatGPT detected 62-68% of bugs (vs. 32% for the best traditional tool) but had a 91% false positive rate, worse than any traditional tool tested.
QASecClaw: A Multi-Agent LLM Approach for False Positive Reduction in Static Application Security Testing
A 2026 study introduced QASecClaw, a multi-agent LLM system that uses a Filter Agent to review SAST findings and classify them as true or false positives, aiming to reduce false positive rates in static application security testing.
Using LLMs as AI Agents to Identify False Positive Alerts in Security Operation Center
A 2024 study used the Llama LLM as an AI agent in a SOC to automatically identify false positive alerts by extracting contextual data, enriching it with threat intelligence, and running it through human-designed investigation playbooks.
Combating the Challenges of False Positives in AI-Driven Anomaly Detection Systems and Enhancing Data Security in the Cloud
A 2024 review of AI-driven anomaly detection in cloud environments found that deep learning models significantly outperform traditional methods, achieving lower false positive rates and higher accuracy, especially when contextual data is integrated.
Do Users Write More Insecure Code with AI Assistants?
A 2023 user study found that participants using an AI code assistant wrote significantly less secure code than those without, and were more likely to believe their code was secure, indicating an overconfidence effect.