SFT-then-RL is All You Need: How Silent Bugs Created a "Mixed-Policy" Illusion
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
This paper debunked the superiority of "mixed-policy" optimization methods for LLM reasoning, revealing that their reported gains over the standard SFT-then-RL pipeline were artifacts of silent bugs in popular training frameworks. By fixing these bugs, the authors demonstrate that a standard SFT-then-RL approach significantly outperforms all existing mixed-policy methods (e.g., LUFFY, ReLIFT) on math benchmarks using models like Qwen2.5 and Llama-3.1.
TL;DR
A new paper from ETH Zürich and EPFL reveals that the alleged superiority of "mixed-policy" methods (which blend SFT and RL) for LLM reasoning is actually a mirage caused by critical bugs in DeepSpeed and OpenRLHF. Once these bugs—which silently dropped gradients and corrupted loss calculations—are fixed, the "old school" sequential SFT followed by RL pipeline dominates the state-of-the-art across all major math benchmarks, proving to be more accurate and significantly more compute-efficient.
The "Mixed-Policy" Hype and the Sparse Reward Problem
In the quest to replicate the reasoning prowess of models like OpenAI’s o1 or DeepSeek-R1, researchers recently moved toward "Mixed-Policy" optimization. The logic seemed sound: RL suffers from sparse rewards (if the model can’t solve a hard math problem, it learns nothing), while SFT alone only leads to memorization. By interleaving them (e.g., LUFFY, ReLIFT, SRFT), researchers claimed they could "bootstrap" the model more effectively.
However, this paper asks a devastating question: Were the baselines we were comparing against fundamentally broken?
The "Silent Killers": DeepSpeed and OpenRLHF Bugs
The authors identified two technical "landmines" that systematically deflated SFT performance in prior works:
- The DeepSpeed Optimizer Bug: In configurations using CPU-offloaded Adam (common for ZeRO-2), a bug caused the system to only send the gradients of the first micro-batch to the CPU. If you had 4 gradient accumulation steps, 75% of your learning signal was silently discarded.
- The Loss Aggregation Bug: A "mean-of-means" error where losses were averaged per mini-batch rather than per token. Since SFT involves variable-length sequences, this incorrectly weighted data, leading to unstable training.
Figure 1: Notice how the buggy DeepSpeed optimizer results in suppressed gradient norms and high loss variability compared to the corrected verl baseline.
Methodology: Restoring the Standard Pipeline
The authors compared a correctly implemented SFT RL (GRPO) pipeline against five major mixed-policy methods.
The SFT stage used high-quality traces from DeepSeek-R1 (OpenR1-Math-46k), followed by Reinforcement Learning from Verifiable Rewards (RLVR). They tested this on both Qwen2.5-Math-7B (math-heavy pre-training) and Llama-3.1-8B (general-purpose).
Results: A Total Wipeout
The findings were conclusive: when SFT is done right, the model enters the RL phase already capable of generating many correct solutions. This provides the "dense" reward signal needed for RL to thrive immediately.
- Performance: On Qwen2.5-Math-7B, SFT RL hit an In-Distribution (ID) average of 57.0, beating the best mixed-policy method (SRFT) by nearly 4 points.
- The Llama Gap: On Llama-3.1-8B, the gap was even more dramatic (+22.2 points). Mixed-policy methods failed because they tried to learn "knowledge" and "reasoning" simultaneously, while sequential SFT provided the necessary knowledge base first.
Table 1: The corrected SFT baseline alone is often stronger than the "improved" mixed-policy results reported in previous papers.
Efficiency: 50 Steps to SOTA
Perhaps the most striking finding is the efficiency. Because the SFT-initialized model is already so strong, the RL phase doesn't need 500 steps. A truncated 50-step RL run achieved 55.6 points—still outperforming all mixed-policy methods while using roughly half the FLOPs of LUFFY or ReLIFT.
Critical Insight: The "Framework Fragility" Problem
As a Senior Editor, I find the meta-contribution of this paper equally important as the math results. The AI community has become heavily reliant on a few "black-box" frameworks (DeepSpeed, TRL, Llama-Factory). When a silent bug enters these foundations, it can invalidate an entire sub-field of research.
Takeaway: Before you implement a complex new "Mixed-Policy" architecture to solve a problem, ensure your "Simple" baseline isn't just a victim of a buggy optimizer.
Conclusion
This work simplifies our understanding of LLM post-training. You don't need complex interleaved blending strategies; you need a high-quality SFT stage followed by a focused RL refining period. The "Mixed-Policy" era may have been a brief detour caused by faulty engineering rather than superior theory.