[arXiv 2025] OAPL: Embracing Off-Policy RL to Supercharge LLM Reasoning

TL;DR

The industry-standard for LLM reasoning—GRPO (used in DeepSeek-R1)—is fundamentally hindered by its "on-policy" requirement, which breaks during massive distributed training. OAPL (Optimal Advantage-based Policy Optimization with Lagged Inference policy) flips the script by embracing off-policy data. It replaces complex importance sampling with a simple squared regression loss, achieving 3x better sample efficiency in coding and superior stability in math, even when the data is 400 steps "stale."

The "Dirty Secret" of Modern LLM Training

In theory, Reinforcement Learning for LLMs assumes the model is learning from its current self. In practice, this is a lie. Due to asynchronous pipelines and hardware-level differences between training kernels (e.g., HuggingFace) and inference engines (e.g., vLLM), the data used for updates is off-policy by the time it reaches the trainer.

Previous attempts to fix this, like Importance Sampling (IS), introduce massive variance that leads to "training collapse" or requires slowing down the inference engine to stay "in sync." The authors of OAPL ask a bold question: Is being on-policy even necessary?

Methodology: The Power of Closed-Form Stability

OAPL moves away from policy gradients and toward KL-regularized RL. The core intuition is that the optimal policy ${\pi }^{\ast }$ has a known relationship with the optimal value function $V^{\ast }$ and the sampling policy ${\pi }_{vllm}$:

$\beta \ln \frac{{\pi }^{\ast }(y|x)}{{\pi }_{vllm}(y|x)}=r(x,y)-V^{\ast }(x)$

By estimating $V^{\ast }$ using a group of rollouts (similar to how GRPO calculates baselines but mathematically more rigorous), the authors derive a squared regression objective:

${\min }_{\pi }\sum {\left(\beta \ln \frac{\pi (y|x)}{{\pi }_{vllm}(y|x)}-ext{Advantage}^{\ast }\right)}^2$

Why this works:

1. No Importance Ratios: It eliminates the $\frac{\pi }{{\pi }_{old}}$ terms that blow up gradients.
2. Natural KL Constraint: The loss itself forces the trainer to stay near the inference policy, preventing the "entropy collapse" (where the model becomes a one-trick pony) often seen in GRPO.
3. Asynchronous Freedom: The trainer can keep updating while the inference engine generates data in the background, only syncing every $L$ (e.g., 50 to 400) steps.

Figure 1: Comparison of OAPL vs. GRPO across AIME-25 and HMMT math benchmarks.

Experimental Showdown

1. Math: Stability vs. Collapse

In competitions like AIME 2025, OAPL didn't just reach higher accuracy; it stayed there. As shown in the entropy curves, GRPO's entropy plummeted (indicating a loss of reasoning diversity), while OAPL maintained a healthy distribution.

Figure 2: Pass@1, 5, and 10 training dynamics. OAPL (blue) is consistently higher and more stable than GRPO (orange).

2. Coding: Efficiency is King

Training coding models is expensive. OAPL matched the performance of DeepCoder (a SOTA GRPO-trained model) on LiveCodeBench while requiring 3x fewer samples. This efficiency comes from the ability to reuse data even as the weight discrepancy grows.

Figure 3: OAPL achieves SOTA accuracy with significantly fewer training generations compared to DeepCoder/GRPO variants.

Critical Insight: The Return of Test-Time Scaling

A common criticism of RL for LLMs is that it only "sharpens" the distribution—meaning it helps the model pick the best answer it already knew, but doesn't improve the chance of finding a needle in a haystack (Pass@k for large k).

The authors debunk this. They show that OAPL improves Pass@k across the entire spectrum (up to $k=256$). By maintaining higher entropy and a more principled objective, OAPL actually improves the model's fundamental reasoning capacity, not just its "vibe."

Conclusion & Future Outlook

OAPL proves that the "on-policy" dogma in LLM post-training is a bottleneck. By shifting to an off-policy regression framework, we get:

• Scale: Train on thousands of GPUs without worrying about millisecond-level synchronization.
• Stability: No more "PPO/GRPO collapse" when kernels differ.
• Efficiency: Reuse data and minimize compute waste.

The future of post-training lies in principled off-policy methods. OAPL is a massive step toward making large-scale reasoning models accessible and robust.

Limitations: While OAPL is highly robust to lag, it still requires periodic syncing. Future work could investigate "fully offline" OAPL using diverse human or external datasets.