Are repository-level coding assistants reliable enough for production software maintenance?

How does AI-generated code actually hold up in production?

The most direct evidence comes from a large-scale empirical study that analyzed over 1,000 files and approximately 3,200 changes from 100 popular repositories on GitHub [2]. The study found that AI-generated files receive less frequent maintenance than human-authored code, and when updates do occur, they affect only a small fraction of the file size [2]. This sounds good on the surface — less maintenance needed — but the type of maintenance tells a different story.

The critical finding is that the most common modifications to AI code are feature extensions, while human-authored code updates focus on bug fixes [2]. This suggests AI code may be structurally sound for initial functionality but lacks the robustness that comes from human debugging and edge-case handling. Even more telling, human developers perform the vast majority of this maintenance themselves [2], meaning the AI assistant isn't reducing long-term maintenance burden — it's shifting the type of work humans must do.

Can developers trust what the AI generates?

A mixed-methods study with 30 programmers directly tested whether AI code completion tools could help developers spot their own errors [3]. The researchers compared three conditions: showing the AI's code alone, highlighting tokens with the lowest generation probability, and highlighting tokens most likely to be edited by a human. The results were striking: highlighting tokens based on generation probability — the most common approach in current tools — provided no benefit over showing the code with no highlighting at all [3]. In contrast, highlighting tokens predicted to need human editing led to faster task completion and more targeted edits, and programmers preferred it [3].

This means that the uncertainty signals built into most AI coding assistants today are not actually helping developers catch problems. The study also found that programmers want highlights that are granular, informative, interpretable, and not overwhelming [3] — a design challenge that current tools largely fail to meet. Combined with the maintenance data from [2], this suggests that even when AI assistants try to flag uncertainty, they're not doing it in a way that improves real-world reliability.

Does the productivity boost come at a cost?

One study confirms that AI assistants significantly boost developer productivity [1], which aligns with the widespread adoption of these tools. However, the same study notes that reliability checking is essential for a holistic understanding of AI-generated code [1]. The productivity gains are real, but they don't automatically translate to reliable production software.

The outsourcing literature provides a useful parallel: a systematic review found that software maintenance offshoring involves complex trade-offs between benefits and risks, with critical success factors that must be carefully managed [4]. Similarly, relying on AI for code generation introduces a new set of maintenance risks that organizations must actively manage. The evidence suggests that while AI assistants can accelerate initial development, the long-term maintenance burden — particularly the need for human-led bug fixes and the poor performance of current uncertainty highlighting — means production reliability still depends heavily on human expertise.

About These Sources

This answer is built on 4 studies (2 peer-reviewed, 2 preprints) — published from 2021 to 2026, 2 from 2024 or later, 1 in Q1–Q2 journals — selected as the most relevant from 4 studies that passed quality screening, drawn from 26 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions