Dense Supervision, Sparse Updates: Decoding the Geometry of On-Policy Distillation

TL;DR

Is On-Policy Distillation (OPD) just a faster version of Supervised Fine-Tuning (SFT), or is it something else entirely? This paper reveals that despite receiving "dense" token-level feedback from a teacher, OPD behaves like a surgical strike: it modifies a tiny, specific subnetwork (often <20% of weights) and avoids the "principal directions" of the original model.

Key Insight: It is the on-policy nature of the data (the model learning from its own mistakes) that dictates this sparse geometry, not the sparsity of the reward signal itself.

The Missing Link in Post-Training

Modern LLM post-training typically follows two paths:

1. SFT/Distillation: Dense labels, but off-policy data (fixed datasets). This usually results in massive, dense parameter rewriting.
2. RLVR (Reinforcement Learning): On-policy data, but sparse rewards (binary success/fail). This results in sparse, localized updates.

OPD sits in the middle: it uses on-policy data but dense labels. Does it produce dense or sparse updates? By analyzing "checkpoint deltas" ($\Delta W=W_{trained}-W_{src}$), the authors find that OPD is much closer to RL than to SFT.

Methodology: Peering into the Weight Space

The authors analyzed 10 model pairs (including Qwen, MiniCPM, and Vision-Language Models) using three lens:

• Coordinate Sparsity: What percentage of weights actually changed? (Answer: Very few).
• Spectral Concentration: Is the update "low-rank"? (Answer: It’s full-rank but the energy is concentrated in a few directions).
• Source Geometry: Does it change the "important" weights of the base model? (Answer: No, it shifts weights that were previously near-zero).

Figure 1: Layerwise update sparsity for DS-Qwen OPD. Most projection matrices remain substantially sparse throughout the network.

Core Findings: The "Off-Principal" Signature

The most striking discovery is the Off-Principal Movement. In linear algebra terms, every weight matrix has "principal directions" (singular vectors) that define its current knowledge. OPD updates effectively ignore these directions, instead writing into the "null space" or low-magnitude coordinates.

As shown above, OPD updates are nearly 100x smaller in magnitude than traditional distillation and significantly more sparse.

Functional Proof: The Subnetwork is Enough

To prove this sparsity isn't just numerical noise, the authors performed a "Subnetwork Intervention." They identified the 17.5% of coordinates that changed during OPD and then re-trained the model from scratch, but only allowed those specific 17.5% to move.

Figure 2: Subnetwork-masked training (Green) recovers nearly identical reasoning performance (AIME accuracy) compared to full training (Blue).

This confirms that the "Task Vector" of reasoning knowledge is naturally sparse. We don't need to update the whole model; we just need to find the right "lottery ticket" subnetwork.

The Optimizer Paradox: Why We Still Need AdamW

Previous research suggested that for sparse RL updates, simple SGD might be enough. However, this paper finds that OPD still requires AdamW.

Why? Because even though the final update is sparse, the dense teacher signals create high "gradient heterogeneity." The second-moment scaling in AdamW helps navigate this complex gradient landscape, even if the model ends up only moving a few weights in the final result.

Critical Analysis & Conclusion

Takeaway

OPD is essentially "Adaptive On-Policy Editing." It doesn't rewrite the model's core identity; it fine-tunes a sparse subnetwork using the model's own distributional language.

Limitations

• Static Analysis: The paper looks at final checkpoints. We still don't fully understand the trajectory of how these sparse masks emerge during training.
• Model Scale: While tested on 1B-7B models, it remains to be seen if 100B+ models exhibit the same extreme off-principal sparsity.

Future Outlook

This work paves the way for OPD-native adaptation. Instead of standard LoRA (which is dense in its low-rank space), we could use "Sparse-LoRA" or "Orthogonal Fine-Tuning" (OFT) that specifically targets these off-principal regions, leading to even more efficient post-training for the next generation of reasoning models.