[ArXiv 2024] RLSD: Curing the Information Leakage of Self-Distilled Reasoning Models

TL;DR

The AI community is currently obsessed with "Scale and Reason," using Reinforcement Learning (RLVR) to bake "Thinking" into models. However, standard RL is slow (sparse rewards), and Self-Distillation is unstable (information leakage). RLSD (RLVR with Self-Distillation) solves this by using the teacher model NOT as a target to mimic, but as a magnifying glass to identify which specific tokens in a reasoning chain actually drove the correct answer.

The "Shortcut" Trap: Why OPSD Fails

On-Policy Self-Distillation (OPSD) seems like a free lunch: let the model look at the answer (privileged info) to act as a teacher for itself (when it doesn't have the answer).

However, the authors prove a "Trilemma" and an Irreducible Mutual Information Gap. Because the teacher knows the answer, its token distribution is fundamentally different. If you force the student to match that distribution (KL Divergence), the student eventually learns to "cheat" by encoding correlations to the answer in its weights.

• The Symptom: A model saying "Based on the reference solution above..." when there is no reference solution in the prompt.
• The Result: Performance peaks early and then collapses as the model stops reasoning and starts "leakage hunting."

Figure 1: OPSD (red) crashes as leakage intensifies, while RLSD (blue) keeps climbing.

Methodology: From "What to Say" to "How Much it Matters"

The core insight of RLSD is decoupling. In standard GRPO, every token in a correct sequence gets the same "+1" reward. This is "coarse."

RLSD uses the teacher to calculate an Evidence Ratio: $w_t={\left(\frac{P_{Teacher}(y_t)}{P_{Student}(y_t)}\right)}^{sign(A)}$

1. Direction Anchoring: The Environment Verifier (is the answer right?) decides if we reward or punish.
2. Magnitude Modulation: The Self-Distillation Teacher decides which tokens get the most credit. If a specific step in a math problem aligned perfectly with the ground truth, it gets a higher weight ($w_t$). If it was just filler text like "Therefore," it gets a neutral weight.

Figure 2: The RLSD Workflow — Environment rewards provide the 'compass', Self-distillation provides the 'acceleration'.

Experimental Results: Faster and Sharper

RLSD was tested on the Qwen3-VL-8B architecture across grueling benchmarks like MathVista and MMMU.

• Superior Efficiency: RLSD reaches the performance of 400 steps of GRPO in just 200 steps.
• Higher Ceiling: By fine-tuning the credit assignment, the model learns more robust reasoning paths than the "uniform" reward of standard RL.
• Heatmap Evidence: Visualizing token credit shows that RLSD correctly identifies critical "pivotal" reasoning steps (e.g., the actual calculation) vs. generic narration.

Table 1: RLSD consistently outperforms standard RL (GRPO) and Self-Distillation (OPSD) baselines.

Critical Insight: The End of Behavioral Cloning?

For years, distillation meant "matching the teacher's logits." RLSD suggests that for Reasoning (System 2) tasks, behavioral cloning is actually dangerous because it encourages shortcuts. Instead, we should treat teachers as Information Filters.

This work establishes a critical framework for future "R1-style" models: use the verifier for the "Truth" and the model-internal states for the "Credit."

Limitations

While RLSD effectively stops leakage, it still requires a "Reasoning Trace" or at least a "Ground Truth" for the teacher pass during training. The next frontier will be applying this "Magnitude vs. Direction" logic to tasks where even the verifier is subjective or learned.

Final Takeaway

RLSD is a "drop-in" replacement for GRPO that effectively doubles training efficiency by simply asking the model: "Now that you know the answer, which part of your previous rambling actually helped you get there?"