[NuMuon] Training for Compressibility: Bridging the Gap Between Full-Rank Optimization and Low-Rank Deployment

TL;DR

Optimization and deployment have long been at odds: while "full-rank" optimizers like Muon accelerate LLM pretraining, they often produce weight matrices that struggle with aggressive compression. NuMuon resolves this by introducing a nuclear-norm-constrained update to the Muon optimizer. By forcing updates to be low-rank during training, it produces models that retain their intelligence even when 80% of their singular values are discarded.

Background: The Muon Paradox

Recent advances in LLM pretraining have highlighted Muon, an optimizer that orthogonalizes momentum updates to treat all singular directions uniformly. This leads to remarkably fast convergence. However, there is a catch: although Muon-trained models surprisingly exhibit an emergent low-rank structure, this structure is "brittle." When you try to compress these models using standard SVD-based pipelines (like SVD-LLM or ASVD), their performance collapses at high compression rates.

The Core Insight: Update-Weight Alignment

The authors identified that the problem isn't just the rank of the weights, but how well the optimizer updates align with the weights'.

• Muon applies updates that are often misaligned with the dominant spectral subspace of the weights.
• NuMuon uses a nuclear-norm constraint to ensure updates are not only low-rank but also stay aligned with the most important directions of the weight matrix.

Methodology: The NuMuon Update

NuMuon reformulates the Muon update as a Linear Minimization Oracle (LMO) over a set that intersects a spectral-norm ball and a nuclear-norm ball.

1. Mathematical Intuition

The nuclear norm is the convex proxy for rank. By limiting the "budget" of the nuclear norm in the update direction $\Delta W$, the optimizer is forced to pick only the most significant $k$ singular vectors.

2. Efficient Implementation

Computing a full SVD every step is too expensive. NuMuon uses a Randomized Block Krylov method to approximate the top-$k$ subspace, making it practical for billion-parameter models.

Figure: The evolution of normalized stable rank under different optimizers.

Experimental Battleground

The researchers tested NuMuon on Qwen3, Olmo2, and Llama3 architectures (0.6B to 1.8B parameters).

Key Findings:

• Convergence: NuMuon matches Muon's aggressive training speed, significantly outperforming AdamW.
• Compressibility: At 40% compression, NuMuon-trained Llama3-1.8B models show a 55.8% improvement in downstream task averages over Muon.
• Stable Rank: NuMuon induces a strictly lower stable rank across all layers, essentially "baking in" compressibility during the pretraining phase.

Figure: Validation perplexity vs. compression rate. NuMuon (purple) remains stable where others fail.

Critical Analysis & Conclusion

NuMuon is a significant step toward compression-aware training. Instead of training a model and then "fixing" it via post-training compression, NuMuon ensures the model is born ready for low-resource environments.

Limitations & Future Work:

While NuMuon is highly effective, the choice of rank $k$ is currently managed by a scheduler. Future research could explore automated rank adaptation, where the model decides how much rank each specific layer needs based on its gradients' tail energy.

Takeaway

If your goal is to deploy high-performance LLMs on edge devices or memory-constrained servers, NuMuon proves that the optimizer is your most powerful tool for ensuring compressibility without the usual accuracy penalty.