[CVPR 2026] DYPO: Harmonizing the SFT-RL Dilemma with Dynamic Policy Optimization

TL;DR

Post-training LLMs usually involves a delicate dance between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). SFT is "safe but biased," while RL is "innovative but volatile." DYPO (Dynamic Policy Optimization) breaks the traditional sequential pipeline by introducing instance-level routing. It uses an ensemble of teachers to fix SFT's bias and a novel Group Alignment Loss (GAL) to tame RL's variance, leading to massive gains in complex math reasoning and out-of-distribution (OOD) tasks.

The Core Conflict: Bias vs. Variance

The "SFT-then-RL" paradigm is the current industry standard, but it's fundamentally flawed. SFT forces the model to mimic specific teacher distributions (Fitting Bias), limiting its ability to find better, novel reasoning paths. When we switch to RL, the sparse reward signals and stochastic sampling lead to "Exploration Traps" (High Gradient Variance), where the model struggles to converge.

Existing "unified" methods tried to fix this with simple loss weighting (e.g., $Loss=\lambda L_{sft}+(1-\lambda )L_{rl}$). However, the authors of DYPO argue that this is a "surface-level" fix. You cannot solve a statistical conflict (Bias vs. Variance) by just mixing scalar values; you need a structural solution.

Methodology: Divide and Conquer

DYPO's brilliance lies in its Dynamic Difficulty Grading. Instead of treating all samples the same, it samples a group of responses ($k=8$) and routes them based on the outcome:

1. Easy: The model solves it every time. These are discarded to save compute—they offer no new learning signal.
2. Hard: The model fails every time. Here, RL is useless because there's no reward to follow. DYPO triggers Multi-Teacher Distillation, using an ensemble of models (like DeepSeek-R1 and Qwen) to provide a robust, low-bias supervised signal.
3. Mid: The "Learning Frontier." The model gets some right and some wrong. This is the sweet spot for RL, but standard GRPO is too noisy. DYPO introduces Group Alignment Loss (GAL) to stabilize the update.

Why GAL Works?

Unlike standard RL advantages that can be unbounded and volatile, GAL uses a sigmoid-based contrastive loss on on-policy rollouts. It "pulls" the model toward successful traces and "pushes" it away from failures within the same group. Mathematically, it acts as an adaptive regularizer that dampens the high-variance noise of RL exploration.

Experimental Results: Precision and Generalization

The methodology was tested across multiple benchmarks (AIME, MATH-500, GPQA). The results reveal two major insights:

1. Surpassing Multi-stage Pipelines: DYPO consistently beat LUFFY, SRFT, and CHORD. On AIME 2025 (one of the toughest math benchmarks), DYPO achieved a 28.7% score, significantly ahead of traditional SFT+RL.
2. OOD Robustness: Perhaps the most impressive result is the performance on GPQA-Diamond (PhD-level science questions). DYPO improved by 16.7% over the base SFT model, proving that the model actually learned reasoning logic rather than just memorizing templates.

Gradient Stability

A key visual proof of DYPO’s efficacy is the Gradient Norm analysis. Standard GRPO (red line) shows extreme volatility, resembling a chaotic system. DYPO (blue line) maintains a smooth, controlled trajectory, which allows for higher learning rates and faster convergence.

Critical Insight & Future Outlook

DYPO moves LLM training closer to how humans learn: we ignore what we already know (Easy), we seek diverse expert help for what we can't grasp (Hard), and we carefully refine our mistakes on topics we are "almost" right about (Mid).

Current Limitations: The reliance on $k=8$ rollouts makes it computationally expensive compared to offline-only methods. Additionally, while it dominates in math and logic, its performance in open-ended creative tasks is still an open question.

The Takeaway: For researchers building the next generation of reasoning agents (o1-like models), the lesson is clear: Dynamic routing is the key to training stability.