[CVPR 2026] SAMPO: Taming the Instability of Agentic Reinforcement Learning

Executive Summary

TL;DR: Training Large Language Models as agents through Reinforcement Learning (ARL) is historically like walking a tightrope—one wrong step leads to a catastrophic "training collapse." ARLArena provides a systematic diagnosis of this instability, revealing that standard token-level clipping is the culprit. By shifting to sequence-level constraints and finer advantage design via the new SAMPO method, researchers can now achieve stable, monotonic improvement in complex, multi-step environments, outperforming frontier closed-source models like GPT-5.2 in task-specific settings.

The Landscape: This work moves beyond "RL tricks" to establish a rigorous academic coordinate system for ARL. It positions itself as a stabilizing foundation, transforming ARL from a "black art" into a reproducible engineering pipeline.

Problem & Motivation: The "Collapse" Phenomenon

In static reasoning tasks (like math), RL has seen massive success (e.g., DeepSeek-R1). However, in agentic settings—where an LLM interacts with a web browser or a physical room—instability reigns supreme.

Why? Because agentic tasks are multi-turn. An "invalid action" at step 2 creates a distribution shift that ruins step 10. This leads to compounding noise and sparse rewards. Prior work often saw the "Success Rate" suddenly plummet to zero mid-training, accompanied by exploding gradient norms. The authors' research intuition was that we were clipping the wrong things: we were looking at individual tokens rather than the coherence of the entire interaction sequence.

Methodology: The Four Pillars of Stability

The authors decompose Policy Gradient for agents into a 4D design space:

1. Loss Aggregation: How we average the "error" across tokens and sequences.
2. Importance Sampling (IS) Clipping: How we prevent the model from changing too fast.
3. Trajectory Filtering: Deleting "useless" data where every attempt failed or succeeded.
4. Advantage Design: How we calculate the "reward" signal for a specific action.

The Innovation: SAMPO

Based on their findings, they proposed SAMPO (Stable Agentic Multi-turn Policy Optimization).

Key Insight: Sequence-level Clipping. Instead of clipping every token's probability change (which allows a whole sequence to still drift too far), SAMPO clips the average change for the entire trajectory. This acts as a "hard brake" on off-policy drift.

Experiments: Stability in Action

The researchers tested SAMPO across ALFWorld (robotics), WebShop (e-commerce), and Sokoban (planning).

Performance Comparison

SAMPO didn't just stop the collapse; it dominated the benchmarks. On ALFWorld, it achieved a 92.7% success rate, whereas standard GRPO struggled at 62.4% and "Tolerant" methods like SAPO collapsed entirely.

The Anatomy of a Collapse

The paper provides a "forensic" look at training failure. They found that collapse is driven by an accumulation of negative-advantage sequences with low IS ratios. Essentially, the model over-learns from "critically bad" examples where its new policy is too different from the old one. SAMPO’s sequence masking and clipping directly neutralize these "gradient bombs."

Critical Analysis & Conclusion

Takeaway: The path to autonomous agents isn't just "more data" or "bigger models." It is about Alignment in the Action Space. By ensuring the RL update doesn't shatter the model's behavioral manifold, SAMPO allows LLMs to scale their reasoning over hundreds of interaction steps.

Limitations: While SAMPO stabilizes training, the "Cold Start" problem remains. The method still requires a small amount of Behavior Cloning (SFT) to get the agent moving in the right direction.

Future Outlook: With the stability bottleneck removed, the next frontier for ARL is Environment Scaling. We can now begin to train agents in environments that are 10x larger and 10x more complex, treating multi-turn interaction as a first-class citizen in the LLM post-training pipeline.