Dense Supervision, Sparse Updates: Decoding the Geometry of On-Policy Distillation

TL;DR

On-Policy Distillation (OPD) is becoming the "third way" in LLM post-training, blending the on-policy benefits of RL with the dense supervision of Distillation. This paper uncovers a surprising paradox: even though the teacher provides feedback for every token, the student model only changes a tiny fraction of its weights. These updates are sparse, spectrally concentrated, and avoid the primary directions of the original model.

The "Identity Crisis" of OPD

In the landscape of LLM training, we usually see two extremes:

1. SFT/Offline Distillation: Large, dense updates that rewrite global parameters using fixed datasets.
2. RLVR (Reinforcement Learning): Extremely sparse updates driven by "win/loss" rewards, often hitting only a specific subnetwork.

OPD sits in the middle. It asks the student to "self-generate" responses (on-policy) but then has a "frontier teacher" (like GPT-4 or a larger Qwen) provide dense, token-by-token corrections. The burning question: Does the dense signal force dense weight changes, or does the on-policy behavior maintain sparsity?

Methodology: Peeking into the Weight Space

The authors analyzed the "Delta" ($\Delta W=W_{trained}-W_{src}$) across various architectures (LLMs and VLMs). They focused on three dimensions:

• Sparsity: How many parameters actually changed?
• Rank/Spectrum: Is the update concentrated in a few dimensions?
• Alignment: Does the update build upon existing features (principal components) or create new, "off-principal" niches?

Table 1: Positioning OPD relative to SFT and RLVR.

Key Finding 1: The Sparse Reality

Across all tests, OPD updates were tiny—relative norms were typically <0.15%. More strikingly, nearly 70-90% of parameters remained effectively untouched (at a $10^{-5}$ threshold).

Interestingly, these updates aren't localized to one layer. They are distributed across the network, but the FFN (Feed-Forward Network) modules typically show the most movement compared to Attention layers.

Key Finding 2: Off-Principal Geometry

The updates don't just "amplify" what the model already knows. The researchers found that OPD updates selectively avoid the principal singular subspaces of the source weights. Instead, they fall on coordinates where the original weights were close to zero.

Intuition: OPD isn't rewriting the model's core logic; it's performing "precision surgery" on the low-magnitude, dormant parts of the model to refine behavioral nuances.

Figure 1: OPD updates are small, sparse, and spectrally concentrated compared to offline distillation.

Interventional Proof: Do These Subnetworks Actually Work?

To prove this sparsity isn't just numerical noise, the authors performed a "Subnetwork Restart":

1. Identify the sparse mask from a completed OPD run.
2. Restart training from the original model, but only allow updates on those masked coordinates.

The Result: Fine-tuning just the 17-22% active subnetwork recovered nearly 100% of the reasoning performance (AIME accuracy) of the full model. This proves that OPD finds a "task vector" that implements specific behaviors without needing the full parameter budget.

Figure 3: Subnetwork-masked training (Green) tracks full training (Blue) almost perfectly, while Random masks (Red) fail.

Why AdamW Still Matters

Previous studies on RL suggested we might only need simple SGD because the updates are so sparse. However, this paper finds that OPD still requires AdamW. Even though the final update is sparse, the dense teacher signal creates a complex gradient landscape throughout training. AdamW's coordinate-wise adaptive scaling remains essential to navigate this "heterogeneous" gradient environment.

Conclusion & Future Outlook

This work shifts our understanding of post-training. It suggests that:

• On-policy data is the driver: The fact that the model is training on its own mistakes is what causes sparsity, not the "richness" of the reward.
• OPD-Native Optimization: We shouldn't treat OPD like standard SFT. There is massive potential for parameter-efficient OPD methods (like LoRA or Orthogonal Fine-Tuning) that specifically target these "off-principal" regions.

In the future, we might see LLM "distillation" that only touches 10% of the weights, leading to faster, more modular post-training pipelines.