[ICLR 2026] Beyond Magnitude: How the Direction of RLVR Updates Unlocks LLM Reasoning

TL;DR

Why does Reinforcement Learning with Verifiable Rewards (RLVR) work so well for LLM reasoning? This paper from the Qwen Pilot Team at Alibaba identifies that the secret lies not in how much the model changes, but in which direction it shifts. By using Δ log p (the log-probability difference), they found that RLVR targets a sparse set of low-probability tokens. They leverage this to create a test-time "booster" and a training-time reweighting strategy that beats current SOTA baselines like DAPO.

The "Magnitude" Trap: Why KL Divergence Isn't Enough

When we compare a base model to its RL-tuned successor, we usually look at Entropy or KL Divergence. While these confirm that RLVR updates are sparse, they are "direction-blind." If you plot a histogram of entropy for a base model vs. an RLVR model, they look identical.

The authors argue that we should look at the signed Δ log p. This metric reveals a bimodal distribution: a positive tail for tokens the RL model loves, and a negative tail for what it has learned to avoid. This isn't just a statistical curiosity; it's the key to identifying the "reasoning-critical" tokens that actually drive performance.

Methodology: Identifying and Exploiting the Shift

1. The Power of Δ log p

Through a Selective Token Replacement experiment, the authors proved that replacing only ~10% of base model tokens at positions with high Δ log p can fully recover the performance of a flagship RLVR model. Magnitude-based metrics (Divergence/Entropy) require much more replacement (up to 30%) to achieve the same effect.

Figure 1: Only Δ log p shows a clear directional shift between base and RLVR models, and it recovers performance faster in replacement trials.

2. The Gradient Intuition: Why Sparse?

Why does RL target such a small subset of tokens? The authors provide a mathematical proof (Lemma 3.1) showing that the gradient norm in policy optimization is proportional to $(1-{\pi }_{h}eta(y))$.

In plain English: The model learns the most from tokens it originally assigned low probability to. These rare tokens receive disproportionately large gradient updates, which explains why the resulting changes are so sparse across the sequence.

3. Exploiting the Direction

The paper proposes two major applications:

• Test-time Extrapolation: During inference, if the model encounters a critical reasoning junction (identified by Δ log p), it doesn't just sample from the RL policy—it "pushes" the distribution even further along that learned direction.
• Training-time Reweighting: During RL, the authors upweight the advantage of low-probability tokens (Probability-Aware Advantage Reweighting). This forces the model to focus on the high-impact "forks in the road" during reasoning.

Experiments & Results

The reweighting method was tested on Qwen2.5-Math-7B and Qwen3-8B-Base. Across AIME and AMC benchmarks, the directional reweighting consistently outperformed the DAPO baseline.

Key takeaways from the experiments:

• Selective Extrapolation surpasses the performance of the original RLVR model by 1-3 points on AIME-24 without any extra training.
• Reweighting leads to longer, more "thoughtful" responses and higher Pass@16 scores.
• Top-p Intervention: A causal experiment showed that filtering out low-probability tokens during RL training destroys reasoning performance, confirming these "minority" tokens are indeed critical.

Critical Insight & Conclusion

This work shifts the paradigm of analyzing LLM RL from "how much did we change the weights" to "which direction are we steering the logits." By identifying that reasoning hides in the tails of the probability distribution, the authors provide a practical toolkit for both more efficient training and smarter inference.

Limitations: The test-time extrapolation requires having both a base and an RLVR model in memory (or using PEFT/LoRA), which increases the compute footprint. However, the performance gains suggest that for high-stakes reasoning, it's a price worth paying.