Does LLM pair programming make code less secure?
Yes, in at least one major study, it did. A 2023 user study found that participants who had access to an AI code assistant wrote significantly less secure code than those who did not [4]. Even more concerning, those with the assistant were more likely to believe their code was secure, suggesting the tool creates overconfidence in flawed code [4]. This is a critical boundary condition: if your priority is security, an LLM assistant may actively undermine it.
However, the same study also noted that the AI assistant improved coding speed [4]. So the tradeoff is real: you may finish faster, but with more vulnerabilities. This finding is especially relevant for developers working on security-sensitive applications.
Can LLMs actually fix code quality issues?
Yes, but only when carefully structured. The CORE tool, which uses a pair of LLMs (one to propose fixes, another to rank them), successfully revised 59.2% of Python files so they passed both static analysis and human review [1]. This is a strong result, but it required a two-LLM architecture and a ranking step to filter out revisions that introduced unintended changes — the ranker reduced false positives by 25.8% [1].
This suggests that a single LLM, used naively, may not reliably improve code quality. The CORE approach shows that with the right scaffolding, LLMs can match or even exceed rule-based automated repair tools (76.8% vs 78.3% fix rate on Java files) with far less engineering effort [1]. So the answer depends on how you deploy the LLM — a solo chatbot is not the same as a structured multi-agent system.
Does the user's skill level matter?
Absolutely. A 2023 study on human pair programming found that different collaboration patterns had different effects on fast-paced versus slow-paced students [2]. For example, 'Guide' and 'Inquiry' patterns significantly improved coding performance for slow-paced students, but not for fast-paced ones [2]. This suggests that the benefit of any pair programming — human or LLM — depends on who you are.
A 2025 study on LLM pair programming found that the human often becomes an 'orchestrator of intent' while the LLM toggles between roles like executor and creative collaborator [5]. This means the quality of the output depends heavily on the human's ability to write good prompts and guide the interaction. If you're a novice, you may get worse results than an expert who can steer the LLM effectively.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2026, 3 from 2024 or later, 1 in Q1 journals, collectively cited 181 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 39 papers retrieved from a database of over 500 million.
Sources used in this answer
CORE: Resolving Code Quality Issues using LLMs
The CORE tool, using a pair of LLMs (proposer and ranker), revised 59.2% of Python files to pass both static analysis and human review, and reduced false positives by 25.8% via the ranker. On Java files, it achieved a 76.8% fix rate, comparable to a specialized program repair tool (78.3%) with much less engineering effort.
Collaborative dialogue patterns of pair programming and their impact on programming self‐efficacy and coding performance
In a study of 46 postgraduate students doing pair programming, four dialogue patterns emerged. 'Guide' and 'Inquiry' patterns significantly improved coding performance for slow-paced students but not for fast-paced ones, showing that the benefit of pair programming depends on the learner's pace.
From Helpful to Trustworthy: LLM Agents for Pair Programming
This doctoral research proposes a systematic study of multi-agent LLM pair programming to externalize intent and use automated feedback for validation, aiming to make LLM outputs more trustworthy and auditable. No empirical results are reported yet.
Do Users Write More Insecure Code with AI Assistants?
In a user study, participants with access to an AI code assistant wrote significantly less secure code than those without, and were more likely to believe their code was secure, indicating overconfidence. This is the most cited paper among these (137 citations).
Developer and LLM Pair Programming: An Empirical Study of Role Dynamics and Prompt-Based Collaboration
An empirical study of developer-LLM collaboration found that roles are dynamic, with humans acting as orchestrators and LLMs toggling between executor, interpreter, and creative collaborator. Prompt design emerged as a critical skill for effective collaboration.