Uni-OPD: Bridging the Reasoning Gap via Dual-Perspective On-Policy Distillation

Executive Summary (TL;DR)

Training smaller "student" models to reason like "expert" teachers is the holy grail of efficient AI. However, standard distillation often fails because students can't explore well, and teachers give bad advice on unfamiliar paths. Uni-OPD introduces a unified framework for LLMs and Multimodal models (MLLMs) that solves this using a dual-perspective recipe. It balances the student's training data for better exploration and "calibrates" the teacher's feedback using a final outcome anchor. The result is a more stable, modality-agnostic reasoner that outperforms its predecessors with fewer training steps.

The "Lying Teacher" Problem: Motivation

In a typical On-Policy Distillation (OPD) setup, the student generates its own reasoning steps, and the teacher provides feedback (rewards) on those tokens. But there is a hidden danger: Unreliable Supervision.

When a student produces a reasoning path that the teacher hasn't seen before (Out-of-Distribution), the teacher's "advice" (log-probabilities) becomes noisy. The authors observed cases where the teacher actually awarded higher scores to incorrect reasoning than to correct ones. Without a global anchor to fix this "order inconsistency," the student essentially learns to hallucinate more confidently.

Methodology: The Dual-Perspective Recipe

Uni-OPD addresses the student's exploration and the teacher's reliability simultaneously.

1. Student Perspective: Exploration Rebalancing

• Offline Difficulty-Aware Balancing: Instead of discarding "easy" or "hard" samples (as in standard RL), Uni-OPD upsamples mid-difficulty tasks to create a uniform training spectrum.
• Online Correctness-Aware Balancing: During training, the system ensures each batch has a balanced ratio (e.g., 1:1) of correct and incorrect student trajectories, preventing the model from collapsing into local optima.

2. Teacher Perspective: Outcome-Guided Margin Calibration

This is the core mathematical innovation. The authors define a Trajectory-level Distillation Return ($G_{OPD}$) and enforce an "Oracle Ordering."

Figure 1: The Uni-OPD framework. Notice how the outcome reward (Correct/Incorrect) acts as a global anchor to shift the teacher's token-level rewards.

The mechanism uses Margin Shift: If the teacher ranks an incorrect trajectory higher than a correct one, the system applies a minimal additive correction ($\lambda$) to restore the margin ($\delta$), forcing the rewards to align with the actual result.

Experiments: Breaking Modality Barriers

The researchers tested Uni-OPD across various settings:

• Multi-Teacher: Merging specialized Math and Code experts into one 4B model.
• Strong-to-Weak: Distilling from a 30B teacher into a 1.7B student.
• Cross-Modal: Distilling from a textual code expert into a multimodal vision-language student.

Table 1: Uni-OPD outperforms standard OPD and other baselines like ExOPD across math and code benchmarks.

Key Results:

1. Efficiency: Uni-OPD reaches higher accuracy with significantly fewer steps than standard Reinforcement Learning (RL).
2. Cross-Modality Reasoning: The study proves that "reasoning" is a modality-agnostic capability. Training on textual code actually improves the visual logic reasoning of a multimodal model!

Visualization: Restoring the Order

The power of margin calibration is best seen in reward heatmaps.

Figure 2: Heatmap of token rewards. Left shows a "confused" teacher rewarding a wrong answer; Right shows the calibrated rewards after Margin Shift.

Conclusion and Takeaways

Uni-OPD demonstrates that the value of a teacher comes from the capability gap, not just its raw size. By ensuring the teacher's signals are consistent with real-world outcomes and forcing the student to explore "the middle ground" of difficulty, we can build smaller, more capable models that excel in both text and vision.

Future Work: The authors suggest expanding this to long-horizon agentic planning and tool-use, where reliable credit assignment is even more critical.