Delightful Policy Gradient

[Google DeepMind] Delightful Policy Gradient: Fixing the Structural Bias of RL Updates

Delightful Policy Gradient

Ian Osband

Summary

Problem

Method

Results

Takeaways

Abstract

This paper introduces the Delightful Policy Gradient (DG), a novel reinforcement learning algorithm that weights gradient updates using a "sigmoid of delight"—the product of advantage and action surprisal (-log probability). DG serves as a robust drop-in replacement for standard policy gradients, achieving SOTA performance in transformer sequence modeling and continuous control without importance ratios.

TL;DR

Standard Policy Gradient (PG) methods are fundamentally biased: they spend too much "gradient budget" on things the model already knows and are easily distracted by rare mistakes. Delightful Policy Gradient (DG) fixes this by gating updates with Delight—a product of advantage and action surprisal. It suppresses "blunders" (rare failures) and prioritizes "breakthroughs" (rare successes), shifting the RL update direction closer to the gold-standard supervised cross-entropy oracle.

The "Easy Task" Trap: Why Standard RL Stalls

In the world of Policy Gradients (REINFORCE, PPO, etc.), update weight is typically determined by Advantage (U). If an action is better than expected, we push its probability up. If it's worse, we push it down.

However, the paper identifies a subtle "pathology":

The Budget Misallocation: Across a batch of different prompts or images, PG allocates more update power to contexts where the model is already successful. If the model is 99% sure on "Easy Task A" and 50% sure on "Hard Task B," Task A actually dominates the gradient. This is the opposite of how we want to learn.
The Blunder Distortion: In a single context, a highly unlikely "bad" action (a blunder) can create a massive gradient spike that distorts the model's parameters, even though the model was already correctly avoiding that action.

Methodology: The Sigmoid of Delight

The authors introduce Delight ( $χ$ ), defined as: $χ = e x t A d v an t a g e im ese x t S u r p r i s a l (- lo g π)$

The update rule then becomes a simple modification. Instead of the standard gradient $g_{t}$ , we use $w_{t} g_{t}$ , where: $w_{t} = σ (χ_{t} / η)$

Why this works (The Intuition):

Breakthroughs (High Delight): An action that was unlikely but yielded a high reward. The gate opens wide ( $w \approx 1$ ). We want to learn from this discovery.
Blunders (Negative Delight): An action that was unlikely and yielded a bad result. The gate closes ( $w \approx 0$ ). The model already knew not to do this; why let it distort the gradient?
Common Sense (Low Delight): Actions the model already favors have low surprisal, so the gate stays around 0.5, allowing standard learning to continue.

Coefficient vs Advantage Figure 1: (a) How the DG gate (red) treats rare vs. common actions compared to standard PG (blue).

Experimental Proof: Better Alignment, Faster Scaling

The paper validates DG across several domains, but the most striking results come from Gradient Geometry and Scaling.

1. MNIST as a Bandit

By treating MNIST as a bandit (only receiving "Correct/Incorrect" rather than the label), the authors show that DG's update direction is mathematically closer to the Supervised Cross-Entropy Oracle than standard PG. This isn't just "variance reduction"—it's a superior update direction that persists even with infinite samples.

2. The Token Reversal Scaling

On sequential transformer tasks, DG demonstrates a smaller scaling exponent than PPO or REINFORCE. As the tasks get harder (longer sequences, larger vocabularies), DG’s advantage over traditional methods grows exponentially.

Scaling with Difficulty Figure 9: DG (red) maintains high performance as complexity increases, while baselines (PPO, PMPO) collapse.

3. Continuous Control

In the DeepMind Control Suite (28 environments), DG matched or exceeded highly-tuned SOTA methods like SAC and MPO. It was particularly effective in "Humanoid Run," where it discovered successful gaits that caused other methods to plateau.

Control Suite Results Figure 10: DG (red) consistently performs at the top of the bench across diverse physical simulations.

Critical Insight & Conclusion

The fundamental contribution of Delightful Policy Gradient is the realization that the "reward-weighted" update rule of standard RL is mathematically different from the "error-weighted" rule of supervised learning.

By using Surprisal to weight the gradient, DG bridges this gap. It provides a simple, theoretically grounded, and empirically superior way to train RL agents—especially in complex, high-dimensional spaces like Large Language Models (LLMs) and robotics where "breakthroughs" are rare and "blunders" are frequent.

Future Outlook: As we move toward larger-scale RLHF for models like GPT-5 or Gemini, the efficiency of the gradient update becomes the primary bottleneck. DG suggests we should stop treating all samples as equal and start focusing on the "Delightful" ones.

Find Similar Papers

Try Our Examples

Search for recent papers that investigate the "directional bias" of policy gradients compared to supervised learning objectives in Reinforcement Learning from Human Feedback (RLHF).
Which studies first identified the "self-reinforcing dynamic" or "easy-task dominance" in gradient-based bandit optimization, and how does Delightful Policy Gradient's solution compare to them?
Explore if the concept of "action surprisal" gating has been applied to sparse-reward exploration or Off-Policy Evaluation (OPE) to improve sample efficiency.

Contents

[Google DeepMind] Delightful Policy Gradient: Fixing the Structural Bias of RL Updates

1. TL;DR

2. The "Easy Task" Trap: Why Standard RL Stalls

3. Methodology: The Sigmoid of Delight

3.1. Why this works (The Intuition):

4. Experimental Proof: Better Alignment, Faster Scaling

4.1. 1. MNIST as a Bandit

4.2. 2. The Token Reversal Scaling

4.3. 3. Continuous Control

5. Critical Insight & Conclusion