Do AI coding agents actually work in real-world settings?
The answer depends heavily on the task. In a tightly scoped, real-world clinical data management task—automatically coding adverse event terms in drug trials—an AI agent framework (using a large language model and retrieval-augmented generation) cut average coding time by 48.7%, from 8.0 to 3.9 minutes per record, while covering 95.8% of terms [1]. This is a clear win for efficiency in a repetitive, rule-based task. But the same study also found that precision against human gold standards was only 70%, meaning nearly a third of the AI's suggestions needed human correction [1]. So even in its best case, the agent is a productivity booster, not a replacement.
In broader software engineering, the picture is worse. An analysis of 4,550 AI-generated pull requests across 81 open-source repositories found that AI agents changed logging code less often than humans in 58.4% of repositories, and when they did, they often produced logs that humans had to fix—72.5% of post-generation log repairs were done by people [2]. This suggests AI agents miss important non-functional requirements like logging, and their output still requires significant human cleanup.
Why do AI coding agents struggle with complex software tasks?
A key reason is that AI agents fail to follow nuanced instructions. In the logging study, explicit natural-language instructions to log properly were rare (only 4.7% of pull requests contained them), and when they were present, agents failed to comply 67% of the time [2]. This indicates a fundamental gap: current AI agents cannot reliably interpret and act on developer intent expressed in plain language, especially for non-functional requirements like logging, error handling, or security.
Another limitation is correctness. A 2022 evaluation of GitHub Copilot using 33 LeetCode problems across four languages found that only 57% of Java suggestions and 27% of JavaScript suggestions passed all provided tests [3]. A separate 2023 study using ChatGPT 3.5 to build a mobile app from scratch concluded that the tool could generate functional code only up to a certain level of complexity, and required an interactive, iterative process with increasingly detailed prompts [4]. This suggests that for novel or complex logic, AI agents still need heavy human guidance and verification.
Is there enough real-world evaluation evidence to trust these tools?
Not yet. While large datasets like CodeNet—with 14 million code samples across 55 languages—exist to train AI for code, they are designed for benchmarking, not for evaluating real-world deployment [5]. The real-world studies we have are limited in scope: one focused only on oncology trial data and a single language pair [1], another on logging behavior in open-source projects [2], and two on narrow coding tasks like LeetCode problems or a single mobile app [3][4]. None of these studies test AI agents in a full, production-grade software development lifecycle with evolving requirements, team collaboration, or legacy code integration.
The evidence also shows that AI agents often produce code that looks plausible but is subtly wrong—for example, relying on undefined helper methods or generating code that could be simplified [3]. Without large-scale, longitudinal studies across diverse real-world projects, it is premature to conclude that AI coding agents are reliable enough for unsupervised use in critical software systems.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2021 to 2026, 2 from 2024 or later, collectively cited 591 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 76 papers retrieved from a database of over 500 million.
Sources used in this answer
Application of Artificial Intelligence in MedDRA Coding: A Practical Exploration from Clinical Data Management Perspective.
In a real-world clinical data management study using adverse event data from pharmaceutical trials, an AI agent framework reduced coding time by 48.7% and achieved 85.1% precision in spelling detection, but overall precision against human gold standards was only 70%, indicating substantial need for human review.
Do AI Coding Agents Log Like Humans? An Empirical Study
An empirical study of 4,550 AI-generated pull requests across 81 open-source repositories found that AI agents changed logging code less often than humans in 58.4% of repositories, failed to comply with explicit logging instructions 67% of the time, and humans performed 72.5% of post-generation log repairs.
An empirical evaluation of GitHub copilot's code suggestions
An evaluation of GitHub Copilot using 33 LeetCode questions across four languages found that Java suggestions had the highest correctness at 57%, while JavaScript was lowest at 27%, and the generated code sometimes relied on undefined helper methods.
Empirical evaluation of automated code generation for mobile applications by AI tools
A study using ChatGPT 3.5 to generate a Flutter mobile app from scratch found that the AI could produce functional code only up to a certain complexity level, requiring an interactive process with increasingly detailed prompts to achieve acceptable results.
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
The CodeNet dataset, containing 14 million code samples across 55 languages, provides a large-scale resource for training AI for code tasks like classification and translation, but it is a benchmark dataset, not a real-world evaluation of deployed AI coding agents.