[SWE-CI] Beyond One-Shot Fixes: Evaluating LLM Agents through the Lens of Continuous Integration

TL;DR

Static benchmarks are no longer enough to measure the true capabilities of AI software engineers. SWE-CI is a new repository-level benchmark that shifts the focus from "one-shot" bug fixing to long-term codebase maintenance. By simulating a Continuous Integration (CI) loop with 100 real-world tasks spanning months of evolution, it reveals a harsh reality: while LLMs are getting better at writing code, they still struggle to prevent regressions and manage technical debt over time.

Problem: The "Snapshot" Trap

In the current LLM evaluation ecosystem (HumanEval, SWE-bench), an agent is given a problem and a single chance to provide a patch. If the tests pass, the agent wins.

However, as any senior engineer knows, passing tests today is not the same as writing maintainable code for tomorrow. A brittle, "hacky" fix might pass a test suite now but break the entire system when the next feature is added. Existing benchmarks cannot see this "invisible" technical debt because they only look at a single snapshot in time. To truly evaluate an AI developer, we must observe how its decisions at Step 1 affect its ability to succeed at Step 10.

Methodology: The CI Loop & Dual-Agent Protocol

To solve this, SWE-CI introduces an iterative evolution paradigm. Instead of a single prompt, the evaluation is a multi-turn process where the agent must evolve a "Base Commit" into a "Target Commit" through dozens of CI cycles.

1. The Architect-Programmer Workflow

The authors emulate a professional environment using two distinct agent roles:

• The Architect: Analyzes failing tests, identifies root causes, and writes a high-level requirement document (limit 5 items) to ensure incremental, manageable progress.
• The Programmer: Interprets the requirements and implements the code changes without seeing the raw test failures directly, preventing the model from simply "gaming" the tests.

2. EvoScore: Measuring Long-Term Value

Standard benchmarks use binary pass/fail results. SWE-CI introduces EvoScore, which uses a decay/growth factor $\gamma$. If $\gamma >1$, the score places more weight on the success of later iterations. This mathematically rewards agents that build a solid foundation early on, making later tasks easier to complete.

Experiments: A Reality Check for SOTA Models

The researchers tested 18 models, consuming over 10 billion tokens. The insights are profound:

• Accelerating Capability: Models released most recently (2026+) show a steep upward curve in maintainability, led by the Claude series.
• The Regression Crisis: Figure 6 highlights a critical weakness. Most models have a "Zero-Regression Rate" of less than 0.25. This means in 75% of projects, the AI eventually broke something that was previously working during the maintenance process.
• Short-term vs. Long-term: Some models (like Kimi and GLM) prioritize immediate test passing, while others (like DeepSeek and GPT-4 series) show better performance when the metric favors long-term stability.

Figure: The evolution of EvoScore across different model families.

Critical Analysis & Conclusion

SWE-CI represents a significant step forward in making AI evaluation match the "messy" reality of software engineering. By forcing agents to live with the consequences of their own code, it exposes the difference between a "coder" and an "engineer."

Takeaway: The next frontier for AI agents isn't just "more parameters" or "larger contexts"—it's architectural reasoning. To survive in a SWE-CI environment, agents must learn to prioritize extensibility and modularity over quick fixes.

Limitations: The benchmark is currently focused on Python. Expanding this to compiled languages (C++/Rust) where CI loops involve complex build systems and longer link times would further test the agents' patience and precision.