Why do SWE-bench scores overstate real coding ability?
The main reason is that SWE-bench and its variants are not clean measures of problem-solving skill. A 2024 analysis of SWE-bench found that 32.67% of successful patches from a top agent (SWE-Agent + GPT-4) involved 'solution leakage'—the answer was effectively written in the issue report or comments [1]. Another 31.08% of passed patches were 'suspicious' because the test cases were too weak to verify correctness [1]. When these problematic cases were removed, the agent's success rate plummeted from 12.47% to 3.97% [1]. This means that a large chunk of apparent success is not genuine engineering ability.
A second, even more troubling issue is memorization. Two independent 2025 studies found that state-of-the-art models could identify the buggy file path from just the issue description—without seeing the repository—with up to 76% accuracy on SWE-bench-Verified, but only up to 53% on tasks from repositories not in the benchmark [2][3]. Similarly, models were 3–6 times better at finding edited files on SWE-bench-Verified than on fresh benchmarks like BeetleBox and SWE-rebench [3]. This gap strongly suggests the models have seen the benchmark tasks during training, so they are recalling answers rather than reasoning from scratch [2][3].
What extra challenges do scientific engineering tasks pose?
Scientific software engineering adds domain-specific hurdles that generic coding benchmarks don't capture. SWE-bench Science, a 2026 benchmark with 119 tasks from 98 repositories across 20 scientific domains, found that even the best agent (Claude Code with Opus-5) solved fewer than half the tasks (pass@1 below 50%) [4]. The authors identified four recurring failure modes: lack of scientific knowledge or abstraction, misguided exploration or surface-level repairs, incomplete repair coverage or system integration, and failure to generalize scientific knowledge to new cases [4].
Interestingly, adding scientific guidance is not always helpful. A paired ablation in the same study showed that well-grounded scientific information can improve performance and token efficiency, but poorly aligned guidance can actually hurt—inducing 'anchoring' on wrong ideas and not improving repair success [4]. This means that for real scientific tasks, agents need both strong coding skills and accurate domain knowledge, and the interaction between the two is delicate.
Are these problems unique to SWE-bench Science?
No—they are systemic across the entire SWE-bench family. The same data-quality issues (solution leakage, weak tests) were found in SWE-bench Lite and SWE-bench Verified [1]. A separate 2025 study using automated test augmentation found 345 erroneous patches incorrectly labeled as passed in the original SWE-bench, affecting 40.9% of SWE-bench Lite and 24.4% of SWE-bench Verified leaderboard entries [5]. This shows that the benchmark's test suites are often too weak to distinguish a correct fix from a superficial one.
The benchmarks are also static and aging. SWE-bench and its variants have not been updated since release, cover a narrow set of repositories, and rely on manual curation—which increases the risk of overfitting and data contamination [6]. Newer benchmarks like SWE-bench-Live, which uses GitHub issues created since 2024, show a substantial performance gap compared to static benchmarks, even under controlled conditions [6]. This suggests that agents perform worse on fresh, unseen tasks, which is exactly what you'd expect if they were memorizing older benchmark data.
About These Sources
This answer is built on 6 studies (1 peer-reviewed, 5 preprints) — published from 2024 to 2026, 6 from 2024 or later — selected as the most relevant from 7 studies that passed quality screening, drawn from 23 papers retrieved from a database of over 500 million.
Sources used in this answer
SWE-Bench+: Enhanced Coding Benchmark for LLMs
A manual analysis of SWE-bench found 32.67% of successful patches from SWE-Agent+GPT-4 involved solution leakage and 31.08% were suspicious due to weak tests; filtering these dropped the resolution rate from 12.47% to 3.97%.
The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
Diagnostic tasks showed models achieve up to 76% accuracy in identifying buggy file paths from issue text alone on SWE-bench-Verified, but only up to 53% on non-SWE-bench repositories, indicating memorization or data contamination.
Does SWE-Bench-Verified Test Agent Ability or Model Memory?
Claude models performed 3 times better on SWE-bench-Verified than on BeetleBox and SWE-rebench, and 6 times better at finding edited files, suggesting the models may have seen SWE-bench-Verified tasks during training.
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
SWE-bench Science, with 119 tasks from 98 repositories across 20 scientific domains, found the best agent (Claude Code with Opus-5) achieved pass@1 below 50%, and identified four failure mechanisms including scientific knowledge deficits and surface-level repairs.
UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
UTBoost, an automated test augmentation framework, identified 345 erroneous patches incorrectly labeled as passed in SWE-bench, affecting 40.9% of SWE-bench Lite and 24.4% of SWE-bench Verified leaderboard entries.
SWE-bench Goes Live!
SWE-bench-Live, with 1,319 tasks from GitHub issues created since 2024, shows a substantial performance gap compared to static benchmarks, highlighting the risks of overfitting and data contamination in older benchmarks.