Rethinking OPD: Why Your Stronger Teacher is Failing Your Student

TL;DR

On-Policy Distillation (OPD) is the "secret sauce" behind many SOTA models like Qwen3 and GLM-5, providing dense rewards that are far more sample-efficient than outcome-based RL. However, it is notoriously fragile. This paper reveals that a teacher's benchmark score is a poor predictor of success. Instead, the secret lies in Thinking-Pattern Consistency. If the student and teacher don't "think" alike, the distillation signal becomes noise.

Background: The "Free Lunch" of Dense Supervision

Unlike traditional Off-Policy distillation (where a student learns from fixed teacher data), OPD lets the student generate its own answers (rollouts) and asks the teacher: "What would you have said at this exact moment?" This provides a reward signal for every single token, effectively solving the "exposure bias" problem. But as the researchers found, this "free lunch" has a steep price if the pairing is wrong.

The Anatomy of Failure: Phenomology & Mechanism

The authors identify two "Laws of OPD":

1. Thinking-Pattern Consistency: The student and teacher must share a compatible candidate space. If the student’s top-k tokens don’t overlap with the teacher’s, the gradient signal vanishes.
2. Genuinely New Knowledge: A larger model from the same family (e.g., 7B vs 1.5B) often provides no gain because they were trained on the same data. They are "scale-up" copies, not "knowledge-up" upgrades.

Key Evidence: The Shared Token Hook

The researchers found that successful OPD is driven by a "virtuous cycle" in the Overlap Region.

• The 99% Rule: A tiny set of shared tokens (the top-k overlap) accounts for 97-99% of the total probability mass.
• Sufficient Statistics: Optimizing only these shared tokens achieves the same results as optimizing the entire vocabulary.

Figure 1: The Overlap Ratio is the heartbeat of OPD. When it stagnates (bottom curve), distillation fails.

A Practical Recipe for Success

If your OPD run is failing, the authors suggest two surgical interventions:

• Off-Policy Cold Start: Before turning on OPD, perform Supervised Fine-Tuning (SFT) on the student using teacher-generated rollouts. This "warms up" the student to the teacher’s thinking style.
• Teacher-Aligned Prompts: Use the exact prompt templates the teacher was trained on. Even small changes in the prompt (e.g., "Think step by step" vs "Reason carefully") can break thinking-pattern consistency.

Figure 2: The impact of an off-policy cold start. Initializing with SFT (top curve) provides a higher performance ceiling and more stable training.

The "Trajectory Ceiling": The Limits of OPD

Perhaps the most critical insight is the Trajectory Depth problem. As a student's response grows longer, it drifts further away from states the teacher "understands."

• Reward Degradation: At a prefix length of 1K tokens, the teacher has a massive accuracy advantage. At 16K tokens, that advantage evaporates to near zero.
• Back-to-Front Instability: Noise starts at the end of the response and propagates backward, eventually destabilizing the entire model.

Critical Insight & Conclusion

This paper shifts the paradigm from "Bigger Teacher is Better" to "Compatible Teacher is Better." It proves that OPD is not a universal knowledge transfer tool but a high-precision alignment mechanism.

Takeaway for Practitioners: When selecting a teacher for distillation, don't just look at the MMLU or AIME leaderboard. Look at the Overlap Ratio on your student's rollouts. if the overlap is below 70%, your distillation is likely doomed without a "cold start" or a shift in prompt alignment.

Figure 3: Post-trained teachers (RL) provide more transferable "new knowledge" than simple same-family scale-ups.