Rethinking On-Policy Distillation: Why Your Stronger Teacher Might Be Failing Your Student
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
The paper investigates the training dynamics of On-Policy Distillation (OPD) for Large Language Models, identifying why stronger teachers sometimes fail to improve students. It introduces a systematic diagnostic framework and demonstrates that OPD success depends on "thinking-pattern consistency" and "new knowledge," achieving SOTA-level reasoning improvements when these conditions are met.
TL;DR
On-Policy Distillation (OPD) is the "dense reward" secret sauce behind recent reasoning models like Qwen3 and GLM-5. However, it is notoriously fragile. This paper reveals that OPD success isn't about the teacher's benchmark score—it's about thinking-pattern consistency and new knowledge. By analyzing token-level dynamics, the authors show that OPD works by aligning a tiny slice of high-probability tokens and offer a "recipe" to fix failing distillation runs.
The "Strong Teacher" Paradox
In traditional Supervised Fine-Tuning (SFT), we assume a smarter teacher always produces a better student. In OPD—where the student generates its own answers and the teacher provides per-token feedback—this rule breaks. The authors found cases where a 7B teacher failed to improve a 1.5B student, while a weaker teacher from the same family succeeded.
Why? The research points to two critical conditions:
- Thinking-Pattern Consistency: If the teacher and student don't "think" alike (measured by top-k token overlap), the distillation signal becomes noise.
- Genuinely New Knowledge: If the teacher and student were trained on the same data, the teacher might have higher scores but no "new" insights to give.
Methodology: Looking Under the Hood
The authors investigated the token-level mechanism of successful versus failing runs.
1. Progressive Alignment
Successful OPD is characterized by a "virtuous cycle." As training progresses, the student's top-k tokens begin to match the teacher's. Interestingly, these "overlap tokens" account for 97%–99% of the probability mass.
2. Overlap Sufficiency
Through a brilliant ablation study, the authors proved that you don't need the full vocabulary log-probs. Optimizing only the shared tokens matches the performance of standard OPD. Non-overlap tokens—where the models disagree fundamentally—actually contribute very little to the gradient.
The Recovery Recipe: How to Fix Failing OPD
If your distillation is stagnating, the paper suggests two practical fixes:
- Off-policy Cold Start: Run a quick SFT phase on teacher-generated data before starting OPD. This "warms up" the student to the teacher's thinking style.
- Teacher-Aligned Prompts: Use the exact prompt templates and data distributions the teacher was trained on. This sharpens the signal.
A Warning on Long-Horizon Reasoning
Perhaps the most sobering finding is that reward quality degrades with trajectory depth. As the student generates longer and longer reasoning chains (Chain-of-Thought), it drifts into "unfamiliar territory" for the teacher. By the time the response hits 10k+ tokens, the teacher's feedback often becomes noisy or destabilizing. This suggests current OPD methods have a "horizon ceiling" for complex, multi-step agentic tasks.
Critical Analysis & Conclusion
This paper shifts the OPD narrative from "dense rewards are better" to "aligned rewards are mandatory."
- Takeaway: When picking a teacher, prioritize architectural and family lineage over raw benchmark leaderboards.
- Limitations: The study focuses heavily on math reasoning. Whether these dynamics hold for creative writing or coding remains to be seen.
- Future Work: The "reward degradation" at long horizons points to a need for hybrid rewards—token-level for short-term logic and outcome-level (RL衡) for global correctness.
In the quest for smaller, faster reasoning models, OPD remains our best tool—provided we respect the delicate geometry of the reward landscape.