[2026] SWE-rebench V2: Scaling Multi-Language Software Engineering Agents via RL-Ready Environments

TL;DR

The bottleneck for creating a truly autonomous AI Software Engineer isn't just the model—it's the data. While we have plenty of code on GitHub, we don't have enough executable, reproducible training environments. SWE-rebench V2 changes the game by releasing a pipeline that harvests 32,000+ executable tasks across 20 languages, specifically designed to power the next generation of Reinforcement Learning (RL) for SWE agents.

The Motivation: Moving Beyond "Python-Only" Benchmarks

Current SWE agents are improving rapidly, with recent jumps driven by RL. However, RL requires thousands of trials. If your environment setup fails or your test signals are "noisy" (flaky), the agent learns nothing.

The authors identify three core challenges:

1. Heterogeneity: Every language (Go, Rust, Java, etc.) has its own build system.
2. Setup Fragility: Static installation scripts fail as often as they work.
3. Oracle Quality: Many GitHub issues are too vague for an AI to solve without "cheating" by looking at the solution patch.

Methodology: The Interactive Setup Agent

Instead of a rigid script, the authors use an Interactive Setup Agent. Think of it as a DevOps-specialized LLM that lives in a Docker container. It tries to install dependencies, reads the error logs, and "fixes" the environment until the tests actually run.

The Construction Funnel

The pipeline follows a sophisticated 5-step process:

1. Mining: Sourcing PRs from GitHub Archive.
2. Setup Synthesis: The Qwen3-powered agent discovers the build/test commands.
3. Execution Validation: Running tests before and after the fix to ensure the "fail-to-pass" (F2P) signal is valid.
4. LLM Judging: Using an ensemble of models (GPT-OSS, GLM, DeepSeek) to filter out "vague" issues.
5. Metadata Enrichment: Tagging tasks with labels like IMPLICIT_NAMING or EXTERNAL_DEPENDENCY.

Key Technical Insight: Why This Works

The breakthrough in SWE-rebench V2 is the interactivity. As shown in the authors' ablations, a non-interactive pipeline stays stuck at a 15.7% success rate for repository setup. By allowing an agent to "try, fail, and retry," the success rate jumps to 62.7% (using Qwen3-480B with 10 attempts).

Experiments: Frontier Models Face the "Wild"

The researchers tested seven frontier models, including Claude Opus-4.5 and GPT-5.2 (Medium), across five languages.

Top Findings:

• Language Gap: Performance is significantly higher in Python than in compiled languages like Scala or Go. This highlights the "Inductive Bias" of current models toward high-resource ecosystems.
• Failures aren't always model errors: The authors identified "B-category pathologies." For example, Test Suite Coupling (B1), where a model writes a correct fix but breaks a fragile, unrelated test, provides a complex but valuable signal for "regression avoidance."

Critical Analysis & Future Outlook

SWE-rebench V2 is the most comprehensive effort to date to turn "GitHub history" into a "World Model" for software engineering. By providing pre-built Docker images, they've lowered the entry barrier for RL researchers.

Limitations:

• Single Container Constraint: The current pipeline struggles with multi-service systems (e.g., projects requiring a separate Redis or Postgres instance).
• Reward Signal: While test-based oracles are the "gold standard," they don't capture non-functional requirements like code style or latency, which are still crucial for real-world production.

Conclusion

SWE-rebench V2 isn't just another benchmark; it's a training substrate. By scaling to 20 languages and 32,000+ tasks, it provides the industrial-scale data required to bridge the gap between "coding assistants" and "autonomous software engineers."

For those interested in the raw data, the authors have released the datasets, collection code, and associated container images.