The Confidence Trap: Why Self-Distillation Can Sabotage LLM Reasoning
Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
This paper investigates why self-distillation, a popular post-training paradigm for Large Language Models (LLMs), can paradoxically degrade mathematical reasoning performance. Through extensive experiments on models like Qwen3 and DeepSeek-R1, the authors identify that the suppression of "epistemic verbalization"—the model's explicit expression of uncertainty (e.g., "Wait", "Hmm")—is the primary cause of out-of-distribution (OOD) performance drops of up to 40%.
TL;DR
More concise isn't always better. A new study reveals that self-distillation (where a model learns from its own solution-guided "teacher" version) often kills the "Wait..." and "Hmm..." moments in an LLM's thought process. This suppression of uncertainty—termed epistemic verbalization—leads to a catastrophic failure in out-of-distribution (OOD) mathematical reasoning, with performance plummeting by up to 40% on benchmarks like AIME.
The Paradox of the "Perfect" Teacher
In the current LLM post-training landscape, self-distillation is a rising star. By letting a model observe the correct answer (acting as a teacher) and then training its "student" self to reach that answer more quickly, researchers have successfully made models faster and more concise.
However, the authors of this paper noticed a disturbing trend in mathematics: as the models got "smarter" (higher training scores) and "faster" (shorter responses), they actually got dumber when faced with new, unseen problems.
Figure 1: While self-distillation works for Chemistry (a), it causes a performance collapse in Math (b) despite shortening response lengths.
Epistemic Verbalization: The "Secret Sauce" of RL
Why does this happen? The researchers point to Epistemic Verbalization. Strong reasoning models (like DeepSeek-R1) aren't just calculating; they are "talking to themselves." They use markers like:
- "Wait, let me re-check..."
- "Actually, if we consider..."
- "Hmm, that doesn't seem right."
These aren't just "filler" tokens. They represent the model's ability to navigate uncertainty. When a teacher model is given the ground-truth solution, it skips these steps because it already knows the path. When the student mimics this "perfect" shortcut, it loses the internal machinery required to handle doubt when the answer isn't known.
Methodology: Measuring Information Richness
The researchers tested four levels of guidance, from "Unguided" to "Full Solution Guided," and found a direct correlation: the more information the teacher has, the more "confident" and "concise" the reasoning becomes, and the fewer epistemic tokens it produces.
Figure: Teacher-guided traces are significantly shorter but strip away the critical 'thinking' process.
When they fine-tuned models on these "concise" traces (Dsg) versus "long, uncertain" traces (Dug), the results were stark. SFT on concise, solution-guided data crashed the AIME24 score from 54.7% to a measly 20.2%.
Task Coverage and the Scaling Law of Doubt
One of the paper's most profound insights is the relationship between Task Coverage and Generalization.
- If you are training a model on a very narrow task (e.g., balancing 100 specific chemistry equations), self-distillation is great. It removes redundant fluff and optimizes for speed.
- If you are training for General Reasoning (thousands of diverse math problems), the model needs those uncertainty tokens to bridge the gap to unseen problems.
Figure: Comparing GRPO (Standard RL) and SDPO (Self-Distillation). SDPO forces a length reduction that eventually chokes off performance gains.
Conclusion: Keep the "Hmm"
The industry's current obsession with "concise reasoning" might be a double-edged sword. This research proves that:
- Confidence is local, uncertainty is global: Removing doubt helps on known tasks but kills adaptability.
- Reward Correctness, Not Style: Post-training objectives must be careful not to penalize the "thinking" tokens that enable self-correction.
- Fixed vs. Moving Teachers: Using a "moving target" teacher in self-distillation creates a feedback loop of over-confidence, further degrading robustness.
For developers of the next generation of O1-style "reasoning" models, the message is clear: Efficiency at the cost of epistemic depth is a recipe for OOD failure.