How could harder software engineering change software engineering agents over the next two years?

Why harder benchmarks are the real driver of agent improvement

The biggest force pushing software engineering agents forward is the arrival of benchmarks that reflect real, long-horizon work—not just isolated coding puzzles. SWE-Bench Pro, introduced in 2025, contains 1,865 problems from 41 actively maintained repositories, with tasks that can take a professional engineer hours to days and often require patches across multiple files [9]. That's a stark contrast to earlier benchmarks that were simpler and more easily gamed. The authors explicitly designed it to be contamination-resistant and to capture enterprise-level complexity, which means agents can no longer rely on memorized snippets or single-file fixes.

This shift matters because it changes what 'good' means. On the older SWE-bench Lite, a simple agentless approach (Agentless) achieved 32% accuracy—the highest among open-source agents at the time—by just localizing, repairing, and validating patches without any complex tool use [2]. But on the harder SWE-Bench Pro, even the best agents fail on most tasks, and the authors clustered failure modes to show that current models struggle with long-horizon reasoning and multi-file changes [9]. So the bar is rising, and agents that were impressive on toy tasks are now being exposed as inadequate for real-world software engineering.

Training methods are catching up: better rewards and guidance

One of the biggest bottlenecks has been training agents in complex environments where rewards are sparse—an agent might take dozens of steps before it knows if its patch works. A 2025 framework called Agent-RLVR tackles this by adding 'agent guidance'—hints like high-level plans and error feedback—during training, then using reinforcement learning from verifiable rewards (RLVR) to update the policy. The result: on SWE-bench Verified, it lifted Qwen-2.5-72B-Instruct's pass@1 (the chance the first attempt fixes the issue) from 9.4% to 22.4%, and further to 27.8% when the guided data was also used to train a reward model [1]. That's a 3x improvement from a single training change, showing that smarter training signals can unlock capability without needing a bigger model.

Another approach, SWE-Dev, scales up training data by synthesizing test cases and agent trajectories. Their 7B and 32B models achieved 23.4% and 36.6% success rates on SWE-bench Verified, outperforming other open-source agents at the time [4]. And a third method, from Alibaba, focuses on scaling test-time compute—letting the model 'think longer' rather than making it bigger. Their 32B model hit 46% on SWE-bench Verified, beating DeepSeek R1 671B and OpenAI o1, which are 20x larger [5]. The common thread: these are all training or inference strategies that make better use of existing model capacity, not just bigger models.

The catch: agents still need humans and structure

Despite these gains, the evidence is clear that agents are not ready to work unsupervised. A 2026 interview study with 17 experienced developers found that oversight is central to successful human-agent collaboration—developers engage in a priori control, co-planning, real-time monitoring, and post hoc review, and they struggle with reviewing agent-generated code, often using test results as a proxy for correctness [7]. This is not a minor detail; it's a fundamental part of how agents will be used in practice.

The technical papers reinforce this. On the Linux kernel, a 2025 study found that state-of-the-art agents achieve only 41.6% top-1 accuracy at file-level fault localization—meaning they often can't even find the right file, let alone fix the bug [3]. And while multi-agent systems like AgileCoder (which mimics Agile methodology with roles like Product Manager and Tester) outperform earlier systems like ChatDev and MetaGPT on code generation benchmarks, they still rely on structured workflows and dynamic code graphs to manage complexity [8]. Even the best hierarchical agent design, discovered automatically via bandit optimization (BOAD), only ranks second on SWE-bench-Live with a 36B model, beating larger models but still not solving most issues [6]. So the realistic near-term picture is: agents get better at specific sub-tasks, but humans will still be in the loop, and the most effective systems will be those that impose structure—whether through roles, graphs, or guidance—rather than letting an agent free-wheel.

About These Sources

This answer is built on 9 peer-reviewed studies — published from 2025 to 2026, 9 from 2024 or later, collectively cited 330 times — selected as the most relevant from 10 studies that passed quality screening, drawn from 47 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions