[Pre-print 2026] SD-ZERO: Turning Sparse Binary Rewards into Dense Self-Supervision
Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
SD-ZERO is a novel post-training framework that enables Large Language Models (LLMs) to self-improve on math and code reasoning tasks using only binary outcome rewards. By combining Self-Revision Training (SRT) and on-policy self-distillation, it achieves over 10% performance gains on Qwen3 and Olmo-3 models, outperforming strong baselines like GRPO and RFT.
TL;DR
Researchers from Princeton, CMU, and UofT have introduced SD-ZERO (Self-Distillation Zero), a framework that allows LLMs to master complex math and coding without any external expert demonstrations. By first learning to fix its own mistakes and then distilling that fixing-logic into its first attempts, the model achieves a 10.5% average performance boost while becoming 2x more token-efficient than traditional self-correction methods.
The "Sparse Reward" Bottleneck
In the world of mathematical and code reasoning, we usually have a "verifier" (like a compiler or an answer key) that gives a binary reward: 1 (Pass) or 0 (Fail).
- The Problem: A "0" doesn't tell the model where it went wrong—was it a typo in line 5 or a fundamental logic error in line 20?
- The RL Tax: Standard RL (like GRPO or PPO) spends millions of tokens sampling different paths to stumble upon the right logic.
- The Distillation Gap: Methods that use dense supervision usually need a "teacher" model (like GPT-4o or DeepSeek-R1), which is expensive and creates a dependency on proprietary tech.
Methodology: The Two-Act Play
SD-ZERO breaks the cycle by turning the model into its own teacher through two distinct phases.
Phase 1: Self-Revision Training (SRT)
Instead of just keeping the correct answers (like Rejection Fine-Tuning/RFT), the model is trained to look at a failed attempt, see the "Fail" reward, and generate a correction.
- L_revision: "I failed, here is the fix."
- L_generation: "Here is the answer from scratch."
This phase "primes" the model to understand the relationship between its errors and the final correct output.
Phase 2: On-Policy Self-Distillation
Once the model is a decent "Reviser," the authors frozen it to act as a Teacher. A second copy (the Student) generates new answers. If the student makes a mistake, the teacher looks at the attempt + the reward and provides a token-level probability distribution for the correction. The student then learns to match this "corrected" distribution directly in its first pass.
Key Insights: Token-Level Self-Localization
Why does this work better than RL? The authors found a phenomenon called Token-Level Self-Localization. Even though the reviser only knows the final answer was wrong, its internal attention focuses heavily on the specific tokens that caused the failure.
As seen in the figure above, for incorrect responses (r=0), the KL divergence (the "teaching signal") is concentrated on a small fraction of "faulty" tokens. The reviser effectively points its finger at the exact moment the logic diverged.
Experimental Battlecard
SD-ZERO was tested on competitive benchmarks (AIME, LiveCodeBench) against heavyweights like GRPO.
- Performance: +10.5% improvement on Qwen3-4B-Instruct.
- Efficiency: It uses roughly the same token budget as GRPO but delivers significantly higher accuracy.
- Length: While SRT models tend to "ramble" and backtrack, the Distillation phase teaches the model to internalize the logic, cutting response length by 50%.
Internalizing the "Wait..." Moment
One of the most fascinating aspects of this research is how the model's behavior evolves. In the SRT phase, models often use phrases like "Wait, let me start over." Through distillation, the SD-ZERO model learns to anticipate these pitfalls. It stops needing to "start over" because it has integrated the correction logic into its initial thought process.
Limitations & Future Work
SD-ZERO currently thrives in verifiable domains where a binary reward is ground-truth. The authors note that extending this to "Thinking Models" (like DeepSeek-R1 or OpenAI o1) is non-trivial, as long chains of thought contain "productive exploration" that shouldn't always be penalized as errors.
Conclusion
SD-ZERO marks a shift away from "more data" toward "better loops." By proving that a model's own revision capability can replace an external human or AI teacher, it opens the door for truly autonomous self-improving agents in math, science, and engineering.