How much can repository-level coding assistants actually improve security?
They can roughly double the detection rate of traditional static analysis tools, but with a major trade-off in false positives. In a 2023 study comparing 11 static analyzers against ChatGPT on 92 PHP vulnerabilities from the OWASP top 10, ChatGPT found 62-68% of flaws, while the best traditional tool managed only 32% [1]. Even combining multiple traditional analyzers only reached 53%—still below ChatGPT's performance [1]. This means a team using an AI assistant could catch many more vulnerabilities early, especially in languages like PHP that lack built-in security mechanisms.
However, the same study found ChatGPT's false positive rate was 91%, meaning only about 9 in 100 alerts were real vulnerabilities [1]. For context, the worst traditional tool had an 82% false positive rate [1]. So while AI finds more real bugs, it also floods developers with far more false alarms—potentially overwhelming teams if used as a standalone solution.
Can false positives be controlled without losing detection power?
Yes—by combining AI assistants with runtime verification or static analysis, teams can slash false positives while keeping high detection rates. A 2025 study tested a runtime monitoring system (Copilot-RV) on 1,247 AI-generated C programs with 18 types of seeded vulnerabilities [2]. The monitors detected 94.3% of injected flaws with only 2.1% false positives and a median runtime overhead of 3.7% [2]. This is a dramatic improvement over the 91% false positive rate of ChatGPT alone [1].
The runtime approach works by checking code behavior against formal safety rules (e.g., memory safety, API usage) during execution, rather than just scanning the code statically [2]. When combined with static analysis, this hybrid pipeline reduced manual verification effort by 73% compared to static analysis alone [2]. So the evidence strongly suggests that false positives are manageable—but only when AI is paired with a second, more precise detection layer.
What are the caveats—and when do these tools fall short?
The main caveat is that AI assistants still struggle with complex, repository-level code. A 2025 benchmark (A.S.E) evaluating leading LLMs on real-world programming tasks found that current models perform well on isolated code snippets but their security drops significantly when dealing with multi-file, repository-level scenarios [3]. Larger reasoning budgets (more compute) did not reliably produce more secure code [3]. This means that while AI can help with local vulnerability detection, it may miss context-dependent flaws that span multiple files or functions.
Another nuance: not all false positives are truly useless. A 2021 study argued that many static analysis warnings labeled 'false positives'—like integer overflow alerts—actually point to real weaknesses that should be fixed for reliability, even if they don't cause immediate crashes [4]. So teams should carefully triage AI alerts rather than dismissing them all as noise. The bottom line: repository-level assistants improve security best when used as part of a layered strategy—combining AI detection with runtime monitoring or static analysis—and when teams invest in tuning alert thresholds for their specific codebase.
About These Sources
This answer is built on 4 studies (3 peer-reviewed, 1 preprint) — published from 2021 to 2025, 2 from 2024 or later — selected as the most relevant from 4 studies that passed quality screening, drawn from 52 papers retrieved from a database of over 500 million.
Sources used in this answer
New Tricks to Old Codes: Can AI Chatbots Replace Static Code Analysis Tools?
In a study of 92 PHP vulnerabilities, ChatGPT detected 62-68% of flaws—double the best traditional static analyzer's 32%—but had a 91% false positive rate, versus 82% for the worst traditional tool [1].
Securing AI-Generated Code with Runtime Verification
A runtime verification system (Copilot-RV) on 1,247 AI-generated C programs detected 94.3% of seeded vulnerabilities with only 2.1% false positives and 3.7% runtime overhead; combining it with static analysis reduced manual verification effort by 73% [2].
A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
A repository-level benchmark (A.S.E) found that current LLMs struggle with secure coding in multi-file scenarios, and larger reasoning budgets do not reliably improve security [3].
Is a False Positive really False Positive?
The paper argues that many static analysis warnings labeled 'false positives' (e.g., integer overflow) are actually real weaknesses that should be fixed for reliability and security [4].