When can you trust an AI-generated pull request?
AI pull requests are most reliable for well-scoped, repetitive tasks like updating dependencies or generating descriptive summaries. A study of 2,904 JavaScript projects found that Dependabot's automated security pull requests were accepted 65.42% of the time, often merged within a day, and only 3.2% caused build failures [7]. This suggests that for routine dependency updates, AI is highly dependable.
Similarly, GitHub's Copilot for Pull Requests, which auto-generates PR descriptions, led to faster review times and a higher likelihood of being merged across 18,256 PRs [4]. Developers using this tool often added their own manual input to the AI-generated text, showing that the AI works best as a starting point, not a final product [4].
However, even in these success stories, human judgment was still involved. The Dependabot study found that the main reason for rejecting a security PR was not a bot error but a concurrent human modification to the same dependency [7]. This means the AI's suggestion was technically correct, but the human context overruled it.
Where do AI pull requests cause problems?
AI-generated code tends to receive less maintenance over time than human-written code, which is a red flag for production software that needs ongoing care. An analysis of over 1,000 AI-generated files from 100 popular repositories found that these files were updated less frequently, and when they were updated, the changes were mostly feature additions rather than bug fixes — the opposite of human-maintained code [6]. This suggests AI code may introduce hidden technical debt.
Another study found that AI coding agents like Copilot and Devin often required explicit reviewer intervention — 15.4% of merged AI pull requests needed human feedback or direct commits before they could be accepted [1]. This means that even when an AI PR is ultimately merged, it often isn't ready on its own.
Code review agents also have a low signal-to-noise problem: when asked to find all hidden issues, they generate many false positives, which can waste developer time [5]. The study's authors warn that measuring success by 'issues found' alone can obscure the real cost of spurious findings [5].
How should teams use AI pull requests in production?
The evidence points to a hybrid model: use AI for the first draft or for routine tasks, but always require human review. The rejection rate of AI pull requests is misleadingly high — only 35.7% of rejections were due to actual agent failures, while 31.2% were caused by workflow constraints like merge conflicts or CI failures [1]. This means many rejected AI PRs were technically sound but fell victim to process issues.
Automation tools like GitHub Actions can actually increase PR rejection rates and lengthen review times, so adding more automation on top of AI can backfire [3]. The key is to design a workflow where AI suggestions are funneled to a human reviewer who can filter out noise and catch subtle errors.
Finally, because AI coding agents leave detectable behavioral fingerprints — like specific commit message patterns or code structures — teams can use these signatures to track which contributions are AI-generated and apply different review standards [2]. This kind of governance helps maintain code quality while still reaping the speed benefits of AI.
About These Sources
This answer is built on 7 studies (3 peer-reviewed, 4 preprints) — published from 2021 to 2026, 5 from 2024 or later, 1 in Q1–Q2 journals, collectively cited 76 times — selected as the most relevant from 8 studies that passed quality screening, drawn from 37 papers retrieved from a database of over 500 million.
Sources used in this answer
Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study
Analyzed 9,799 human-reviewed AI pull requests and found that only 35.7% of rejections were due to actual agent errors; 31.2% were due to workflow constraints and 33.1% had no clear rationale [1].
Fingerprinting AI Coding Agents on GitHub
Fingerprinted 33,580 PRs from five AI coding agents, achieving 97.2% F1-score in identifying which agent generated the code, revealing distinct behavioral patterns like Codex's unique multiline commit patterns [2].
GitHub Actions: The Impact on the Pull Request Process
Studied 1,489 repositories using GitHub Actions and found that adoption led to more PR rejections, more communication in accepted PRs, and longer time to accept a PR [3].
Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions
Examined 18,256 PRs with AI-generated descriptions and found they required less review time and had a higher likelihood of being merged, though developers often added manual input [4].
CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents
Introduced CR-Bench and CR-Evaluator for code review agents, finding a low signal-to-noise ratio when agents tried to find all hidden issues, with a trade-off between issue resolution and spurious findings [5].
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
Analyzed over 1,000 AI-generated files from 100 repositories and found AI code receives less frequent maintenance than human code, with updates focused on feature extensions rather than bug fixes [6].
On the Use of Dependabot Security Pull Requests
Investigated 2,904 JavaScript projects using Dependabot and found 65.42% of security PRs were accepted, often within a day, with only 3.2% causing build breakages [7].