Rethinking On-Policy Distillation: Why Your Stronger Teacher is Failing Your Student Model

TL;DR

In the world of Large Language Model (LLM) post-training, On-Policy Distillation (OPD)—where a student learns from a teacher's feedback on its own self-generated responses—is often seen as a "free lunch" of dense supervision. However, this paper from Tsinghua University reveals that OPD is surprisingly fragile. A stronger teacher doesn't guarantee a smarter student. Instead, success depends on thinking-pattern compatibility and knowledge novelty. By tracking the "overlap" of token distributions, the authors provide a recipe to fix failing distillation runs and uncover a "length ceiling" where teacher feedback becomes noise.

The Paradox of the "Superior" Teacher

We've long assumed that in Knowledge Distillation (KD), a more capable teacher leads to a better student. The authors challenge this with a striking phenomenon: Reverse Distillation.

When they distilled a 1.5B model backward to its own pre-RL checkpoint, it lost its reasoning gains. More surprisingly, when they replaced that weak teacher with a much larger and stronger 7B model from the same family, the 1.5B student regressed to the exact same point.

The Insight: High benchmark scores do not equal transferable knowledge. If the teacher and student are trained on the same data, they share the same "thinking patterns," leaving no room for the student to grow through OPD, regardless of the teacher's scale.

The Mechanism: Mapping the "Overlap Ratio"

How do we measure if OPD is working? The authors point to the Overlap Ratio—the percentage of tokens that both the student and teacher rank in their Top-K.

Successful OPD follows a "Progressive Alignment" signature:

1. Overlap Rises: The student and teacher's Top-K sets begin to align (e.g., from 72% to 91% overlap).
2. Entropy Narrows: The student matches the teacher’s confidence levels.
3. Sufficiency: The authors proved that optimizing only the tokens that overlap is enough to match full-vocabulary distillation. The signal that matters is concentrated in the top 1-2% of the probability mass.

Methodology: The Recipe for Success

When OPD fails because the "gap" is too wide, the authors propose two practical remedies:

1. Off-Policy Cold Start

If the student's thinking pattern is too far from the teacher's, they can't "communicate" via token-level rewards.

• Solution: First, perform a standard SFT (Supervised Fine-Tuning) on fixed teacher-generated trajectories. This aligns their thinking patterns before starting the dynamic, on-policy phase.

2. Teacher-Aligned Prompts

The prompts used for distillation matter. By using prompts that the teacher model saw during its own training (RL post-training), the teacher provides much sharper, more confident feedback that the student can actually use to bridge the gap.

Limitations: The Trajectory Depth Problem

One of the most critical findings is that OPD has a horizon limit.

• Reward Degradation: As the student generates longer and longer responses (Chain-of-Thought), it drifts into states the teacher hasn't seen.
• The 7K Sweet Spot: Performance peaks at moderate lengths (3K-7K tokens). Beyond 10K, the "overlap ratio" collapses, student entropy spikes, and the teacher’s "advantage" vanishes.

The instability starts at the end of the response and propagates backward, suggesting that for massive, multi-step reasoning tasks, dense token-level rewards might eventually become detrimental.

Conclusion: A New Coordinate System for Distillation

This research shifts the focus of LLM distillation from Teacher Power to Relational Compatibility.

• Don't just pick the biggest model as your teacher.
• Do monitor the overlap ratio during training as a "canary in the coal mine."
• Do use off-policy warmup if your student is struggling to achieve initial alignment.

As we push toward "Thinking Models" (like DeepSeek-R1 or OpenAI’s o1), understanding the local optimization geometry of these token-level rewards will be the key to training smaller, more efficient reasoners.