AGPO: Protecting the Reasoning Boundary in LLMs via Asymmetric Optimization
AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JD
AGPO (Asymmetric Group Policy Optimization) is a novel Reinforcement Learning with Verifiable Rewards (RLVR) method designed by JD.com researchers to enhance LLM reasoning. By employing a negative-dominant reinforcement strategy and intra-group variance scaling, it achieves SOTA performance on benchmarks like MATH and AIME while preventing the common "reasoning boundary shrinkage" where trained models lose the breadth of the original base model.
TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR) is the "secret sauce" behind models like OpenAI-o1 and DeepSeek-R1. However, standard RL often makes models "stiff"—they get better at easy paths but lose the ability to find complex ones. Asymmetric Group Policy Optimization (AGPO) solves this by focusing on punishing errors and rewarding rare insights rather than over-reinforcing what the model already knows.
The "Boundary Shrinkage" Crisis
In the race to hit SOTA Pass@1 (greedy accuracy), we have inadvertently been lobotomizing our models. Recent studies show that after standard PPO/GRPO training, a model's Pass@256 (the ability to find any correct answer in 256 tries) often drops below the original base model.
This is the Boundary Shrinkage problem: the model "collapses" its probability mass onto a few safe paths, losing the diverse reasoning "fringe" that allowed the base model to solve hard problems.
AGPO: Why Asymmetry is the Answer
The authors from JD.com argue that the mechanism of RLVR isn't creating new logic, but filtering existing paths. AGPO implements three key intuitions:
- NSR > PSR: Negative Sample Reinforcement (pruning wrong answers) is safer than Positive Reinforcement because it redistributes probability mass naturally without forcing a collapse.
- Ignore the Trivial: If a model already finds a path easily, stop reinforcing it. This prevents the "oversharpening" of simple logic.
- The "Rare Gem" Reward: Rare correct paths get a higher relative advantage, pushing the model to explore the difficult "long-tail" of reasoning.
The AGPO Objective
Unlike GRPO, which uses standard normalization, AGPO's advantage function is asymmetric:
Figure 1: AGPO vs Baselines. Note how AGPO maintains a stable, linear negative signal (NSR) to ensure consistent pruning.
Experimental Breakthroughs
AGPO was tested on the most rigorous math benchmarks (AIME 2024, OlympiadBench, MATH) and a massive industrial dataset.
1. Mathematical Reasoning
On the challenging AIME-2024, AGPO improved Qwen2.5-Math-7B's accuracy from 13.9% to 30.1%, significantly outperforming PPO and GRPO. More importantly, it was the only method that didn't destroy the model's exploration boundary at .
Key Result: AGPO achieves highest Pass@1 while maintaining Pass@256 coverage.
2. Real-World Impact: JD Search Ads
At JD.com, AGPO was used to optimize a teacher model that labels query-product relevance.
- Offline: Reduced irrelevant ad matches (PIR) by nearly 25% compared to the base model.
- Online A/B Test: Resulted in a +0.50% CPM (Revenue per thousand impressions) and +0.22% CTR, proving that better reasoning directly translates to business value.
Critical Insight: The "Cold Start" Problem
Despite its success, AGPO reveals a fundamental limit of RLVR: If the base model can't find the answer once in its initial samples (Pass@1 ≈ 0), no amount of RL will help. On the AIME-2025 benchmark, AGPO stayed flat because the base model provided no "correct paths" to amplify.
Conclusion
AGPO marks a shift from "brute-force reinforcement" to "intelligent pruning." By acknowledging that the model's latent potential is already there, AGPO focuses on refining the search space. For practitioners, the message is clear: if you want a model that reasons deeply without losing its creative edge, focus on the negatives and the rare positives.
Takeaway: Asymmetry is the key to balancing exploitation (getting it right now) and exploration (retaining the capacity to get hard things right later).