[BeyondSWE] Can Current Code Agents Survive Beyond Single-Repo Bug Fixing?

TL;DR

While AI code agents have reached 80%+ success rates on standard benchmarks like SWE-bench, these tasks are "localized" at best. BeyondSWE shatters this illusion by introducing a benchmark for the "beyond" cases: cross-repo reasoning, domain-specific science (quantum/bio), dependency migration, and blank-slate repository generation. The results? Even top-tier models like GPT-5.2 and Gemini 3 Pro struggle, often failing to break the 45% success ceiling.

Background Positioning: The End of the "Local" Era

Traditional benchmarks assume all the "truth" is within the codebase. In the real world, developers are detectives; they hunt for documentation, trace issues across third-party libraries, and migrate code across breaking API versions. BeyondSWE moves the goalposts from "Bug Fixer" to "Expert Software Engineer."

Problem & Motivation: The Single-Repo Trap

Current code agents are essentially high-performance "patchers." They are great at reading a local function and fixing a logic error. However, they fail when:

1. The knowledge is elsewhere: Resolving a bug in one repo requires looking at a related issue in a different repo.
2. The domain is specialized: Working on qutip (quantum physics) or biotite (bioinformatics) requires more than just Python syntax—it requires scientific intuition.
3. The change is global: Upgrading from NumPy 1.x to 2.x requires a sweeping refactor across dozens of files, not a single-line patch.

Methodology: Resolution vs. Knowledge Scope

The authors structured the benchmark along two axes:

• Resolution Scope: From isolated function repairs to global system generation (Doc2Repo).
• Knowledge Scope: From "within-repo" to "domain-expert" and "open-web" levels.

To evaluate this, they built SearchSWE, an agentic framework that allows a model to browse the web, read documentation, and then return to its Docker terminal to apply what it learned. Crucially, they implemented an Integrity Safeguard: the agent is blocked from accessing the ground-truth GitHub repository to prevent "cheating" via searching for the original fix.

Experiments & Results: The Search-Code Disconnect

The experimental findings were sobering. Even with web access, the models did not consistently improve.

Key Performance Stats:

• Gemini 3 Pro (the leader) reached a 41.82% average success rate.
• GPT-5.2 followed at 36.19%.
• DomainFix was the hardest category, with rates often dropping below 30%.

The Failure of Search

Why didn't search help? The researchers identified a "Search-Code Disconnect":

1. The Information Landscape Gap: Search engines rank human-readable docs high, but agents need raw code artifacts/diffs which are often buried.
2. Temporal Misalignment: A model searches for "Django migration," finds the 5.x docs, but the local repo is pinned to 2.2. The model applies modern "best practices" that crash the legacy system.
3. Semantic Drift: In a niche library, the word "Fixture" might mean something totally different than in general web development, leading the agent down a rabbit hole of irrelevant information.

Deep Insight: Efficiency vs. Frequency

A fascinating takeaway from the behavior analysis: Gemini 3 Pro searched the least (avg 1.0 calls) yet gained the most benefit (+2.0% gain). Conversely, DeepSeek-V3.2 searched the most (avg 5.0+ calls) and actually saw its performance drop by 0.2%.

Conclusion: More searching leads to more noise. The next generation of code agents needs "Domain Discrimination"—the ability to reject high-ranking search results that violate the local semantic context.

Takeaway & Future Work

BeyondSWE is a wake-up call for the industry. Making an agent "smart" isn't just about longer context windows or better search tools; it's about the tight integration of external context with local constraints. The benchmark is now publicly available, providing a much-needed rug-pull for models that have become "too comfortable" with SWE-bench.