DOPD: Conquering the Privilege Illusion in On-Policy Distillation

TL;DR

On-Policy Distillation (OPD) is the current gold standard for transferring capabilities from giant "Thinker" models to compact students. However, naively providing teachers with "privileged information" (like hints or labels) often leads to Privilege Illusion: where the student tries to copy the teacher's "cheating" rather than learning its actual reasoning logic. DOPD (Dual On-Policy Distillation) solves this by dynamically routing token-level supervision based on an "Advantage Gap," ensuring the student only learns what is truly transferable.

The Problem: The "Privilege Illusion"

In modern AI development, we often give teachers a "peek at the answer" (privileged information) to set a higher performance ceiling during training.

The authors identify a critical failure mode: Information Asymmetry. If a teacher performs better simply because it sees a hint that the student won't see at test time, the resulting supervisory signal is "illusionary." If the student tries to fit these tokens too hard, it suffers from Entropy Collapse—its output distribution becomes dangerously narrow, leading to brittle performance and a loss of exploratory behavior.

Methodology: Advantage-Aware Token Routing

DOPD doesn't treat every token the same. It uses a Dual Distillation paradigm that compares two privileged policies (Teacher vs. Student) to calculate the Privilege Advantage Gap (A).

The Four Regimes of Supervision

Based on the gap ($A$) and prediction confidence ($q$), DOPD routes every token into one of four strategies:

1. High A, High $q_T$ (The Gold Mine): The teacher is much better and very confident. Strategy: Strong Full-vocabulary JS Divergence to capture raw reasoning capacity.
2. Low A, High $q_S$ & $q_T$ (Consensus): Both agree. The gap is just info asymmetry. Strategy: Light Top-K KL to maintain stability without over-fitting shortcuts.
3. Low A, Low $q_S$ & $q_T$ (Unreliable Space): Neither knows what's happening. Strategy: Weak self-regularization to prevent policy drift.
4. High A, High $q_S$ (Student Pride): The student is confident but contradicts the teacher. Strategy: Light privileged-student distillation to allow for autonomous exploration.

Experimental Battlefront

The effectiveness of DOPD was tested across 8 LLM benchmarks (MATH500, AIME25, LiveBench) and 8 VLM benchmarks.

• Performance Recovery: DOPD recovered 89.8% of the teacher-student performance gap in LLMs, far exceeding Vanilla OPD (~34%).
• Scalability: Even when distilling an 8B teacher into a tiny 0.6B student, DOPD maintained a 14.1-point gain, whereas Vanilla OPD's gains vanished as the model size gap grew.
• Stability: Unlike self-distillation methods that frequently crash or collapse in entropy mid-training, DOPD's dual-routing provides a "consistency anchor" that keeps the training curve smooth.

Critical Insight: Why This Matters

The most profound finding in this paper is the Token Analysis (Figure 9). The authors visualize exactly which tokens provide value. They found that in complex reasoning chains, the tokens with a "High Advantage Gap" are usually the logical pivots—the points where the actual "thinking" happens. By focusing computational resources on these pivots rather than "connective breadcrumbs," DOPD achieves SOTA efficiency.

Conclusion & Future Directions

DOPD proves that in the era of "Thinking Models," the quality of supervision is more important than the quantity. It moves the field from "Mimic the Teacher" to "Learn the Teacher's Advantage."

Limitations: The method requires an extra forward pass (higher training cost) and relies on high-quality privileged hints from models like GPT-5.4. Future work will likely focus on generating these hints "on-the-fly" to reduce cost while maintaining the precision of the Advantage Gap.