When Errors Can Be Beneficial: Rethinking Imperfect Rewards in Policy Gradient

TL;DR

In the world of Reinforcement Learning from Human Feedback (RLHF), we usually assume that any error in a Reward Model (RM) is bad. However, this paper proves that not all reward errors are equal. By analyzing the optimization dynamics of policy gradients, the authors show that some errors are "benign" (having no effect) and some are even "beneficial"—actually helping the model converge faster to the optimal output by steering it away from "mediocre" traps.

Background Positioning

This work represents a theoretical shift in RM evaluation. While most SOTA benchmarks focus on "Ranking Accuracy," this research enters the coordinate system as a foundational theoretical analysis that explains why high-accuracy reward models sometimes produce mediocre language models.

Problem & Motivation: The Ranking Accuracy Paradox

Current standards for training Large Language Models (LLMs) via RL rely on proxy reward functions. Whether it's a learned RM or a deterministic rule (like unit tests for code), these proxies are rarely perfect.

The industry largely uses Ranking Accuracy (how often the RM prefers the same response a human would) as the gold standard. However, the authors argue this is too coarse. The core insight is that policy gradient optimization is non-concave. Standard metrics ignore the "physics" of the probability mass: how it moves, where it gets stuck, and how a "wrong" reward might actually push the mass in the right direction faster.

Methodology: Categorizing the "Physics" of Reward Errors

The authors utilize linear softmax policies and gradient flow to categorize errors into three buckets:

1. Harmful: These cause "reward hacking" or slow down optimization.
2. Benign: Errors on outputs that already have very low probability or rewards lower than the current average. They simply don't affect the gradient much.
3. Beneficial: Errors that assign lower rewards to mediocre outputs than the ground truth does.

The "Mediocre Trap" Intuition

Why would a "wrong" low reward be good? Theorem 1 (Theoretical Highlight) proves that in a softmax policy, the gradient is scaled by the output probability. A "mediocre" response (better than average but worse than the best) that has high initial probability can exert a "gravitational pull," trapping the model's mass for an arbitrarily long time. By erroneously giving this mediocrity a very low reward, the RM "repels" the probability mass toward the true optimal output.

Figure 1: High-level categorization of reward errors and their impact on ground truth increase.

Experiments & Results

1. Harm-Aware Accuracy (HAcc)

The authors proposed a new metric, HAcc, which does not penalize an RM for ranking a "bad" response over a "good" one, provided the "bad" response is still below the model's current expected reward.

Experiments across Llama-3.2, OLMo, and Qwen models showed that HAcc correlates significantly better with the actual delta in ground truth reward after training compared to standard Accuracy (Acc).

Table: Comparison of various Reward Models and their RewardBench2 scores vs. their actual impact.

2. The Danger of Partial Rewards

In tasks like code generation, should we give "partial credit"? The theory suggests a trap: if a model finds it much easier to be "partially correct," a partial reward will cause it to stall there.

Experimental evidence in Figure 4 reinforces this: in instruction following, rewarding only full correctness (binary 0/1) often leads to higher final success than rewarding partial correctness (0.5/0.5), because the latter traps the model in a "mediocre" state where it only learns the easiest constraint.

Figure 2: Empirical proof that assigning mediocre rewards to suboptimal outputs (Top Row) stalls optimization, while low proxy rewards (Bottom Row) accelerate the path to the optimum.

Critical Analysis & Conclusion

Takeaway

The effectiveness of a Reward Model is policy-dependent. We cannot evaluate anRM in a vacuum; we must evaluate it in the context of the model it is meant to "teach." Errors on improbable or already-discarded branches of the probability tree are irrelevant.

Limitations

• Bandit Assumption: The theory focuses on single-turn (bandit) environments. Multi-turn dialogues introduce state-dependency that might complicate the "beneficial error" dynamics.
• Feature Similarity: The benefit of repelling mediocre outputs depends on the geometric similarity (inner product) between the features of the mediocre and optimal outputs. If they are too similar, the repulsion might hurt the optimum too.

Future Work

The authors suggest adaptive proxy rewards: starting with partial rewards to help the model find the right neighborhood, but switching to binary rewards as the model matures to avoid the "mediocre trap." This provides a rigorous path forward for designing more efficient RLHF pipelines.