What metrics actually matter for AI-generated PRs?
The most direct productivity metric is cycle time—how long a pull request sits before it's reviewed and merged. A year-long enterprise study of 300 engineers using an in-house AI tool found a 31.8% reduction in PR review cycle time, meaning AI-generated PRs moved through the pipeline nearly a third faster [1]. Another study of 18,256 real PRs on GitHub confirmed that AI-assisted PRs require less review time and have a higher likelihood of being merged [2]. These two studies, using very different methods (controlled enterprise cohort vs. open-source observational data), converge on the same conclusion: AI cuts review time and boosts merge rates.
But speed alone can be misleading if quality suffers. The enterprise study also tracked code volume shipped to production, finding a 28% overall increase, with top adopters pushing 61% more code to production [1]. This suggests the time savings translate into real output, not just faster rejections. A smaller case study of 16 mobile developers added a 'ReviewerScore' metric that measures how well AI-generated code integrates with existing codebases, finding that AI helped with technical onboarding and stack switching [5]. The lesson: use a basket of metrics—cycle time, merge rate, code volume, and integration quality—rather than any single number.
The catch: developers don't just accept AI output—they edit it
A common mistake is treating AI-generated PRs as finished products. The GitHub study found that developers using Copilot for PRs frequently complement automated descriptions with manual input—they don't blindly trust the AI [2]. Similarly, the mobile developer study showed that while AI code generation saved time, human reviewers still needed to assess correctness and technical integration [5]. This means you should measure the edit distance between what the AI produces and what gets merged. If developers are rewriting most of the AI output, the productivity gain is smaller than it appears.
Even automated bots like Dependabot, which generate security update PRs, see a 65.42% acceptance rate—high, but not universal. The 34.58% that aren't merged are mostly due to concurrent dependency changes, not bot failures [4]. This tells you that AI-generated PRs still require human judgment about context and timing. A separate study on bot interactions in open-source projects found that bots can cause information overload and disruption, leading to design strategies like a separate interface for bot messages and a mediator bot to summarize actions [6]. The takeaway: measure not just whether a PR is merged, but how much human effort went into editing, reviewing, and deciding to accept it.
How to set up measurement in your team
Start with a baseline before introducing AI tools. The enterprise study used a cohort design, comparing engineers using AI against those who didn't, over a full year [1]. They tracked adoption patterns—engagement grew from 4% in month 1 to 83% peak by month 6, then stabilized at 60% [1]—showing that measurement must account for the learning curve. Without a baseline and a control group, you can't separate the AI effect from other improvements.
Use the PR4Code dataset as a benchmark if you're training or evaluating AI models for full-feature PR generation. It contains 4,508 Java and 8,831 Python curated PRs with metadata and commit histories, designed to assess AI on realistic, multi-step development tasks rather than isolated code snippets [3]. For ongoing measurement, track: (1) cycle time per PR, (2) merge rate, (3) code volume shipped, (4) developer edits to AI output, and (5) reviewer satisfaction (the enterprise study reported 85% satisfaction for code review features [1]). The mobile developer study also showed that AI tools helped with onboarding new team members and switching between Android and iOS, so consider measuring time-to-productivity for new hires [5].
About These Sources
This answer is built on 6 studies (4 peer-reviewed, 2 preprints) — published from 2021 to 2025, 3 from 2024 or later, collectively cited 79 times — selected as the most relevant from 6 studies that passed quality screening, drawn from 43 papers retrieved from a database of over 500 million.
Sources used in this answer
Intuition to Evidence: Measuring AI's True Impact on Developer Productivity
In a year-long enterprise cohort of 300 engineers, AI-assisted PRs showed a 31.8% reduction in review cycle time, a 28% increase in code shipment volume, and top adopters pushed 61% more code to production; adoption grew from 4% to 83% peak over 6 months.
Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions
In an analysis of 18,256 real PRs on GitHub, AI-assisted PRs required less review time and had a higher likelihood of being merged, but developers frequently edited the AI-generated descriptions manually.
PR4Code: A Pull Requests Dataset for AI Code Generation
The PR4Code dataset provides 4,508 Java and 8,831 Python curated PRs with metadata and commit histories, designed to benchmark AI models on full-feature development tasks rather than isolated code snippets.
On the Use of Dependabot Security Pull Requests
In a study of 2,904 JavaScript open-source projects using Dependabot, 65.42% of security PRs were accepted (often within a day), with only 3.2% causing build breakages; non-merged PRs were mostly due to concurrent dependency changes.
Case study: using AI-assisted code generation in mobile teams
In a case study of 16 mobile developers, AI-assisted code generation improved time-to-completion for onboarding and stack-switching tasks, and a new 'ReviewerScore' metric was proposed to measure code integration quality.
Bots for pull requests
In a participatory design study with 32 practitioners, bots in PRs were found to save time but also cause information overload; 22 design strategies were elicited, including a separate interface for bot messages and a mediator bot to summarize actions.