[MAR 2026] Code-A1: Co-Evolving Coders and Testers via Adversarial RL
Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning
The paper introduces Code-A1, an adversarial reinforcement learning framework that jointly optimizes a Code LLM and a Test LLM through co-evolution. By decoupling the models and using GRPO, it achieves SFT-beating performance on benchmarks like HumanEval+ and MBPP+, specifically reaching an average accuracy that matches or exceeds models trained on human-annotated tests.
TL;DR
Reinforcement Learning (RL) has revolutionized code generation, but it has a "data hunger" problem: it needs high-quality unit tests to provide rewards. Code-A1 solves this by pitting two models against each other—a Coder and a Tester. The Coder earns rewards by passing tests; the Tester earns rewards by breaking the Coder's logic. This adversarial dance, stabilized by a Mistake Book mechanism, allows models to surpass human-annotated performance without a single new human test case.
The Collision Dilemma: Why Self-Play is Broken
Existing "Self-Play" methods usually use one model to generate both code and tests. This leads to a fundamental flaw: Self-Collusion.
- The White-Box Trap: If the model sees the code it just wrote, it "cheats" by writing tests it knows the code will pass.
- The Black-Box Limitation: If you hide the code from the tester to prevent cheating, the tests become generic and fail to find subtle, implementation-specific bugs (e.g., edge cases in a specific loop logic).
Code-A1 breaks this cycle by architectural decoupling. By using two specialized agents with opposing objectives, the framework can safely allow the Tester "white-box" access to the Coder’s output without the risk of mutual back-scratching for easy rewards.
Methodology: The Adversarial Engine
The core of Code-A1 lies in its Asymmetric Co-evolution pipeline.
1. The Adversarial Rollout
- Step 1: The Code LLM generates candidate solutions.
- Step 2: The Test LLM inspects the candidate code (White-Box) and attempts to craft adversarial assertions.
- Step 3: Tests are validated against a "Ground Truth" oracle (to ensure the test itself isn't a hallucination).
- Step 4: Rewards are distributed—Coder gets
+for passing, Tester gets+for exposing a failure.
2. The Mistake Book (Persistence of Memory)
Adversarial training is notoriously unstable. If the Tester suddenly gets "smart," the Coder might get overwhelmed and "forget" how to solve easier problems. Code-A1 introduces the Mistake Book, an experience replay buffer that stores historically failed tests. This forces the Coder to maintain backward compatibility—it can't just pass new tests while regressing on old ones.
Experimental Breakthroughs
The results across Qwen2.5-Coder scales (1.5B to 7B) provide three major insights:
Performance > Human Labels
On HumanEval+, Code-A1 achieved 85.21% (7B model), outperforming versions trained on human-annotated "Golden Tests." This suggests that a dynamic, adversarial tester is a better "teacher" than a static human benchmark.
Efficiency: Small Models, Big Brains
The 3B-parameter Code-A1 model achieved a Mul score of 15.29, outperforming the 7B Base model. This proves that adversarial pressure is a more potent scaling law than parameter count when it comes to logic-heavy tasks like unit testing.
Critical Insight: The "ThreeSum" Case Study
The authors highlight a "ThreeSum" problem where the Coder had a subtle bug in handling duplicate triplets.
- Before training: The Tester produced generic tests that missed this.
- After adversarial training: The Tester "learned" to specifically generate inputs with highly repetitive elements (e.g.,
[-2, 1, 1, 1, 1]) to break the Coder’s pointer logic. This curriculum-like progression is the "Secret Sauce" of Code-A1.
Summary & Future Outlook
Code-A1 demonstrates that we are entering an era of "Model-Generated Alignment." By moving away from static datasets and toward dynamic, adversarial environments, we can push LLMs toward authentic robustness.
Limitations: Currently, the system still requires a "Ground Truth" solution to validate if the Tester's adversarial cases are actually correct. The next frontier? Doing this in "Zero-Shot" mode where no ground truth exists, relying purely on cross-consistency between multiple models.
Senior Editor's Note: Code-A1 is a significant step toward self-evolving autonomous agents. The transition from "learning from humans" to "learning from competition" is the clearest path we have toward Superintelligence.