How much faster do developers actually get with an LLM pair programmer?
The most direct evidence comes from a controlled experiment with GitHub Copilot, where developers were asked to build an HTTP server in JavaScript as fast as possible. The group with Copilot finished 55.8% faster than the group without it [1]. That is a massive time saving — more than half the development time cut — but it came from a single, specific task, not from everyday work. The same study also found that the speed boost was larger for developers who were newer to the field, suggesting LLMs may help people ramp up faster [1].
Another study, using micro-level manufacturing data, found that a 1% increase in AI penetration led to a 14.2% increase in total factor productivity [2]. While this is not software-specific, it shows that AI-driven productivity gains can be large and measurable at the firm level, and that the effect comes through value-added enhancement, skill-biased enhancement, and technology upgrading [2]. So speed is real, but it is not the only thing that matters.
Does faster code mean worse code — and do developers like it?
A case study at TiMi studio, a mobile game developer, compared AI-pair and non-AI-pair projects and found that AI-pair programming actually improved code quality and developer satisfaction [3]. Developers reported benefits like time-saving, error-avoiding, skill-learning, and quality-improving [3]. However, the same study also uncovered real downsides: developers questioned the AI's reliability and explainability, reported a lack of trust and communication, and felt a loss of autonomy and motivation [3]. These are not minor complaints — they can undermine long-term adoption and team morale.
Another paper, Jigsaw, highlights a key caution: large language models do not understand program semantics, so they offer no guarantees about the quality of the code they suggest [5]. Jigsaw's approach was to add post-processing steps based on program analysis to improve accuracy, showing that raw LLM output can be unreliable [5]. This means teams should not just measure speed; they must also track bug rates, code review outcomes, and developer sentiment to know if the productivity gain is real or just faster bad code.
What specific metrics should teams track to measure real productivity impact?
Based on the evidence, teams should measure at least three categories. First, task completion time: the Copilot study shows a 55.8% speed gain on a focused task [1], so track time-to-completion for typical tasks before and after introducing an LLM pair programmer. Second, code quality: the TiMi study found improvements in code quality [3], but the Jigsaw paper warns that LLMs can produce incorrect code [5] — so track defect density, code review pass rates, and rework time. Third, developer satisfaction and well-being: the TiMi study found both benefits and challenges like loss of autonomy and motivation [3], so use regular surveys to capture how developers feel about the tool, their trust in it, and their sense of control over their work.
The TiMi study also noted that AI-pair programming offers benefits like skill-learning and possibility-exploring [3], which are harder to quantify but matter for long-term team growth. Teams should consider qualitative measures like developer interviews or retrospective comments to capture these. The key is to avoid a single metric — speed alone can be misleading if quality drops or developers burn out.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2022 to 2024, 2 from 2024 or later, 1 in Q1–Q2 journals, collectively cited 685 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 52 papers retrieved from a database of over 500 million.
Sources used in this answer
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
In a controlled experiment, developers using GitHub Copilot completed an HTTP server task 55.8% faster than those without it, with larger gains for less experienced developers [1].
AI-Driven Productivity Gains: Artificial Intelligence and Firm Productivity
Using micro-level manufacturing data, a 1% increase in AI penetration led to a 14.2% increase in total factor productivity, driven by value-added, skill-biased, and technology upgrading effects [2].
The Impact of AI-Pair Programmers on Code Quality and Developer Satisfaction: Evidence from TiMi studio
A case study at TiMi studio found AI-pair programming improved code quality and developer satisfaction, but also revealed challenges including trust, communication, autonomy, and motivation issues [3].
LLM-aided Pair Programming for Algorithm Tracing
A pilot study with a GPT-4-based tool, TraceCompanion, explored LLM-based pair programming for algorithm tracing in education, gathering initial student perceptions [4].
Jigsaw
The Jigsaw paper showed that large language models can generate code from intent but offer no quality guarantees, and proposed post-processing with program analysis to improve accuracy [5].