[ICLR 2025] Entropy-Preserving RL: Solving the Collapse Problem in LLM Reasoning

TL;DR

Policy gradient algorithms (like GRPO or PPO) often trade away a model's "creativity" or entropy for immediate reward, leading to diversity collapse. This paper identifies that how entropy evolves during training (the journey) matters more than the final result. By introducing REPO (advantage rescaling) and ADAPO (adaptive clipping), the authors prevent this collapse, achieving SOTA on AppWorld and ensuring models remain "trainable" for future tasks.

The "Greedy" Trap: Why Standard RL Fails

In the rush to boost reasoning performance (pass@1), current RL methods often "over-sharpen" the model. They reinforce what the model already knows, making the distribution so narrow that the model loses its ability to explore alternative correct paths.

The authors argue that this isn't just an algorithmic choice—it's often exacerbated by hidden numerical artifacts. For instance, they discovered that using BF16 quantization during training introduces a multiplicative upward bias in importance ratios, effectively creating an accidental "entropy-decreasing" asymmetric clipping that sabotages training stability.

Methodology: Engineering the Entropy Journey

The core contribution lies in two distinct mechanisms for explicit entropy control:

1. REPO (Regulated Entropy Policy Optimization)

Instead of adding a heavy-handed global entropy bonus (which is memory-intensive), REPO modifies the advantage function directly: $A_{REPO}(s,a)=A(s,a)-{\beta }_s\cdot L(s,a)$ Where $L(s,a)$ is the mean-centered log-probability. By subtracting this term, the algorithm essentially "rotates" the advantage space, boosting the rewards for rare, surprising, but correct actions.

2. ADAPO (Adaptive Asymmetric Clipping)

Building on DAPO, which uses static asymmetric clipping, ADAPO introduces a bidirectional controller. If entropy drops below the initial baseline, it widens the upper clipping bound (${ϵ}_{high}$) to allow for more probability increases; if entropy rises too high, it tightens the bounds.

Figure 1: Comparison of entropy trajectories. Note how algorithms that maintain steady entropy (top) correlate with higher final test accuracy (bottom).

Experimental Battleground: AppWorld & AIME

The authors tested their methods on two fronts: AppWorld (complex tool-use) and AIME (high-level math).

SOTA Results

By fixing numerical precision issues (moving from BF16 to FP16) and applying entropy preservation, the team achieved:

• AppWorld Test Normal: 79% (New SOTA).
• AppWorld Test Challenge: 71%.

The "Sequential Learning" Proof

Perhaps the most striking evidence is the Sequential Learning experiment. Models trained with standard GRPO "collapsed"—their entropy hit near-zero. When these models were then asked to learn a new task, they failed because they could no longer explore. In contrast, REPO-trained models maintained their "curiosity" and successfully adapted to new environments.

Figure 4: REPO-R and ADAPO consistently outperform baselines like GRPO by preserving the model's exploratory power.

Critical Insights: The Numerical Shadow

A major takeaway for practitioners is the impact of 16-bit quantization. The paper proves that BF16 rounding systematically favors entropy decrease. By simply switching to FP16 and ensuring full-precision log-probability calculations, the authors found that many "algorithmic" instabilities simply vanished.

Conclusion: Entropy as a First-Class Citizen

This work shifts the perspective of RL post-training from "reward maximization" to "distribution management." If we want agents that can not only solve known problems but also adapt to new ones, we must treat entropy as a first-class citizen in our loss functions.

Future Outlook: The integration of REPO with asynchronous training pipelines could bridge the gap between high-throughput "weakly on-policy" methods and the high-stability "strictly on-policy" methods like RLOO.