[DeepMind] Delightful Policy Gradient: Fixing the "Success Bias" in RL

TL;DR

Standard Policy Gradient methods (PG) are biased: they spend too much time optimizing things they already know how to do. Delightful Policy Gradient (DG) fixes this by weighting updates with "Delight"—a product of advantage and how surprising an action was. By amplifying rare breakthroughs and silencing rare blunders, DG rebalances the learning budget toward harder tasks, outperforming PPO and SAC across bandits, Transformers, and robotics.

The Success Pathology: Why Standard PG Stalls

In the world of Policy Gradients (REINFORCE, PPO), we update our model by multiplying the gradient of the log-probability by the advantage ($Uabla\log \pi$). While intuitive, this creates a self-reinforcing feedback loop.

Consider two tasks in a batch: one that the model solves 99% of the time, and one it solves 50% of the time. Because the successful task generates more positive samples, it consumes the lion's share of the gradient budget. Effectively, the model keeps practicing its "A+" material while ignoring the "C-" topics. This isn't just a variance issue; it's a directional bias that persists even with infinite data.

Methodology: The Architecture of Delight

The authors introduce a simple, scalar gate $w_t$ applied to the standard gradient. The core logic is:

1. Surprisal (${\ell }_t$): How unlikely was this action? ($-\log \pi (A_t)$)
2. Delight (${\chi }_t$): The product of Advantage ($U_t$) and Surprisal (${\ell }_t$).
3. The Gate ($w_t$): A sigmoid $\sigma ({\chi }_t)$ that filters the update.

Figure: The "Delight" gate treats rare successes and rare failures asymmetrically, whereas standard PG is blind to action probability.

Why $Uimesextsurprisal$ works:

• Breakthroughs (Rare Success): High Advantage $imes$ High Surprisal = Large Positive Delight. The gate opens wide ($w\approx 1$), making the model "pay attention" to this rare discovery.
• Blunders (Rare Failure): Negative Advantage $imes$ High Surprisal = Large Negative Delight. The gate closes ($w\approx 0$), preventing a single freak accident from distorting the policy.

Experiments: From Bandits to Transformers

1. The MNIST Diagnostic

By framing MNIST as a bandit task (the agent only gets a "correct/incorrect" signal, not the label), the authors showed that DG shifts the PG direction closer to the Supervised Cross-Entropy Oracle. This proves that DG effectively "recovers" the lost information about task difficulty that standard RL ignores.

2. Scaling Sequence Models

On the "Token Reversal" task, DG showed a smaller scaling exponent than PPO. This suggests that as sequence length and vocabulary size increase (the "Hard" regime), DG's advantage over standard methods actually grows.

Figure: Scaling with task complexity. DG (red) degrades much more gracefully as horizons ($H$) and vocabulary ($M$) increase.

3. Continuous Control

In the DeepMind Control Suite, DG achieved the lowest aggregate regret across 28 diverse environments, even when compared against highly tuned versions of Soft Actor-Critic (SAC).

Figure: Aggregate regret across 28 tasks. DG (red) is remarkably robust, avoiding the catastrophic failures seen in PPO on complex tasks like Humanoid.

Critical Insights & Conclusion

The brilliance of Delightful Policy Gradient lies in its simplicity. It’s a one-line change to the update rule that structurally corrects the gradient geometry of RL.

Key Takeaways:

• Beyond Variance: Most RL researchers focus on variance reduction. This paper argues we should focus on gradient direction.
• Heuristic to Theory: The sigmoid gate isn't just a trick; the authors derive it from an entropy-regularized objective (see Appendix A), giving it a solid variational foundation.
• Future Impact: If this scaling advantage holds for Large Language Models (LLMs) during RLHF, DG could significantly improve how models learn from rare, high-quality human feedback.

Limitations: While DG works impressively in the tested domains, its performance in extremely sparse reward environments (where "delight" is rarely triggered) remains an open area for investigation.