SWE-Together: Evaluating Coding Agents as Interactive Collaborators, Not Just Calculators
SWE-Together: Evaluating Coding Agents in Interactive User Sessions
SWE-Together is a novel multi-turn benchmark composed of 109 repository-level tasks reconstructed from over 11,000 real user-agent coding sessions. It evaluates coding agents using a reactive LLM-based user simulator that replays original user intents and provides interactive feedback, moving beyond static, one-shot evaluations.
TL;DR
Static benchmarks are no longer enough to measure the true utility of coding agents. SWE-Together introduces a multi-turn, interactive evaluation framework derived from 11,260 real-world user sessions. It proves that the best models don't just write better code; they require significantly less "hand-holding" or corrective steering from the user to reach a solution.
Beyond the Static "Prompt-to-Code" Paradigm
For years, we have judged coding agents on their ability to solve a fixed problem statement in one go (one-shot). However, anyone who has used GitHub Copilot or Zed knows that real coding is a conversation. You ask for a feature, the agent misses a constraint, you correct it, the agent refines the code, and so on.
The authors of SWE-Together argue that current benchmarks like SWE-bench are reaching a ceiling. They fail to capture the interaction cost: the human effort required to guide an agent to success.
The Methodology: Reconstructing Reality
Converting messy, real-world chat logs into a reproducible benchmark is a massive technical challenge. The SWE-Together pipeline involves three rigorous stages:
- Deterministic Filtering: Pruning 11,260 sessions down to those with genuine interaction and detectable repository edits.
- Viability Screening: Using an LLM judge to ensure the task doesn't depend on private keys or external live services.
- Sandbox Reconstruction: Building a "reactive user simulator" that doesn't just read a script but reacts to the agent's specific mistakes.
Figure 1: SWE-Together reframes the evaluation from static (top) to interactive sessions (bottom).
The Secret Sauce: Trajectory-Conditioned Simulation
A major innovation here is the anchored simulator. If the agent takes a different path than the original human-session agent, a "fixed" script would break. SWE-Together's simulator monitors the agent's progress and only speaks up when "Intent Coverage" gaps are detected or when a "User Correction" is triggered.
Insights from the Leaderboard
The study evaluated seven frontier models, including Claude Opus 4.8 and GPT-5.5. The results provide a fascinating look at the "soft costs" of AI:
- The Intelligence-Intervention Inverse: There is a nearly linear relationship between model strength and user intervention. Claude Opus 4.8 requires ~36% less corrective steering than MiniMax-2.7.
- The "Hand-holding" Metric: The benchmark introduces User Correction, a weighted count of how many times the user had to say "Wait, that's wrong" or "Try this instead."
- Stability Matters: Models like GLM-5.2 showed higher performance stability across replicates compared to their predecessors, even when raw success rates were similar.
Figure 2: Capability vs. User Correction. Stronger models (top left) require fewer corrections.
Why Reference Scores Aren't 100%
Interestingly, human "reference patches" only score ~78% on the benchmark. This isn't because the humans failed the task, but because the Agentic Rubric Judge also looks for process requirements—like whether the agent explained the change or answered follow-up questions. This highlights that SWE-Together is evaluating the agent as a teammate, not just a code generator.
Conclusion: A New Compass for Agent Development
The era of one-shot coding benchmarks is ending. SWE-Together provides a necessary roadmap for building agents that are truly collaborative. The takeaway for developers is clear: improving a model's "coding skill" is only half the battle; the real value lies in its ability to follow instructions the first time, minimizing the human mental load of correction.
As we move toward "Agentic Workflows," benchmarks that measure human-AI friction will be the ones that define which models we actually use in production.
Limitations
Currently, the simulator cannot "interrupt" an agent mid-turn or see a visual UI. It relies on text-based logs and tool outputs. Future iterations will likely need to incorporate multi-modal feedback to handle front-end development more effectively.