OPSD: The "Compactor" for Modern Reasoning Models

TL;DR

On-Policy Self-Distillation (OPSD) has been touted as a potential successor to Reinforcement Learning with Verifiable Rewards (RLVR). However, this paper reveals a crucial nuance: for long-trace mathematical reasoning, OPSD doesn't teach the model how to solve new problems—it teaches the model how to stop rambling. By applying OPSD specifically to successful reasoning paths, researchers can compress response lengths by nearly 30% without sacrificing a single point of accuracy, turning it into the ultimate post-RL "clean-up" tool.

Problem & Motivation: The Gap Between "Short" and "Long" Thinking

In "thinking-disabled" tasks (standard LLM chat), the teacher has an easy job. If the student is wrong, the teacher provides the correct direct answer, and the student learns the mapping.

In thinking-enabled mathematical reasoning, the trace is long and complex. The authors noticed that when OPSD is applied here, the accuracy gains reported in previous studies often vanished or turned negative.

• The Insight: A hindsight-conditioned teacher (one that knows the final answer) is great at spotting "fluff" and redundant checks in a long trace, but it struggles to suggest a totally different, correct reasoning path that the student doesn't already know.
• The Hypothesis: OPSD is a repetition-penalty mechanism, not a correction mechanism.

Methodology: Isolating Correction from Compression

To prove OPSD doesn't "repair" logic, the authors split training into two groups:

1. Correct-only: Distilling from successful student rollouts.
2. Incorrect-only: Distilling from failed student rollouts.

The core OPSD formula uses a reverse-KL divergence to minimize the gap between the student's distribution and a "Self-Teacher." The teacher is the same model but "privileged" with extra context (like the ground-truth answer or a reflection).

${\mathcal{L}}_{\mathrm{OPSD}}=\mathbb{E}\left[\frac{1}{|y|}\sum D_{\mathrm{KL}}({\pi }_S\parallel {\pi }_T)\right]$

If OPSD were truly a "logic repairman," the Incorrect-only group should show massive accuracy improvements. If it were a "compactor," only the Correct-only group would remain stable while getting shorter.

Experiments & Results: Compression Wins

The results across 7B-8B models (Qwen3, DeepSeek-R1-Distill) were striking.

• The Compaction Effect: Correct-only OPSD consistently preserved or slightly improved accuracy while slashing lengths. For AceReason-Nemotron-7B, it achieved a 3.5 pp accuracy boost and -17.6% length.
• The Failure of Repair: Incorrect-only OPSD was disastrous, dropping accuracy by up to 10 percentage points. The "privileged" teacher simply made the model more confident in its wrong paths, stripping away the "wait," "hmm," and "let me check" markers that might have led to a self-correction.

Further tests involving richer teacher contexts (expert demos) and "signal reinjection" (periodically reminding the teacher of the answer mid-trace) didn't change the fundamental behavior. They just moved the model further along the tradeoff curve toward more aggressive compression.

Deep Insight: A Revised Training Pipeline

This paper fundamentally reshapes where OPSD belongs in the AI lifecycle. Instead of viewing it as a rival to RLVR (like PPO or GRPO), it should be seen as the third pillar of training:

1. SFT: Learn format and basics.
2. RLVR: Expand capability and "reachable" reasoning through trial and error.
3. OPSD: Compact and de-noise. Once the model knows how to solve a problem, use OPSD to make that solution "cheaper" (fewer tokens) and "cleaner" (less redundancy).

Conclusion

OPSD is the "industrial press" of the reasoning world. It excels at taking a correct but messy and expensive reasoning trace and squeezing it into its most efficient form. For developers looking to reduce the serving cost of frontier reasoning models without losing "intelligence," a "Correct-only" OPSD stage is the new gold standard. It prevents the model from "overthinking" while keeping the logic sharp.

Takeaway: Don't ask OPSD to fix your model's logic; ask it to fix your model's verbosity.