Muon is Not That Special: Demystifying Spectral Optimizers with Freon and Kaon

TL;DR

Is the success of state-of-the-art spectral optimizers like Muon due to their elegant "non-Euclidean geometry"? This paper argues no. By introducing Kaon (an optimizer that uses random noise as singular values) and Freon (which operates in mathematically "impossible" quasi-norm regimes), the authors prove that the real driver of performance is a trade-off between gradient alignment and descent potential, paired with a constant optimal step size.

Problem & Motivation: The Geometric Mirage

In the recent race for LLM efficiency, spectral optimizers like Muon and Shampoo have emerged as SOTA contenders, often justified by the theory of Linear Minimization Oracles (LMOs). The narrative was simple: whitening the gradient's spectrum aligns the update with the "natural geometry" of the loss landscape.

However, the authors noticed "cracks in the facade":

1. Performance is hyper-sensitive to tuning.
2. Optimal updates often fall into quasi-norm regimes (c > 0.5) where the triangle inequality breaks, and standard LMO theory effectively collapses.
3. If the geometry is so critical, why do slight deviations often work better?

Methodology: From "Polar Express" to "Chaotic Noise"

The paper systematically deconstructs the spectral update $(G{G}^{o}p{)}^{-c}G$.

1. Freon: The Interpolator

The authors developed Freon, a family of optimizers that uses a novel QDWH-based iterative approximation (Algorithm 4) to stably compute matrix powers. This allows them to sweep the exponent $c$ from 0 (SGD) to 0.5 (Muon) and beyond to 1.0 (Pseudoinverse-like).

Figure 1: Comparison of update spectral transformations. Note how Freon interpolates across the spectrum.

2. Kaon: The Absurd Baselines

To prove that the precise target spectrum doesn't matter, they created Kaon. It uses a chaotic logistic map to assign random singular values to the gradient. If geometry were king, Kaon should fail miserably.

Experimental Results: The Death of Geometric Necessity

The results are provocative. On WikiText-2 and NanoGPT benchmarks:

• Kaon matches Muon: Despite replacing structured singular values with noise, Kaon matches Muon's validation loss.
• Optimal Exponents are "Illegal": For GPT-2, the best exponent is $c\approx 2/3$. This is in the quasi-norm regime, meaning no unitarily invariant norm can explain this update as a steepest descent.

Figure 2: Validation loss vs LR. Kaon (blue) and Freon c=2/3 (green) demonstrate that Muon (orange) is not a unique peak.

Deep Insight: The Two Local Quantities

The authors propose a new lens to view optimization success through a local Taylor expansion, identifying two key variables:

1. Batch Gradient Alignment (${\gamma }_k$): How much of the "true" global gradient direction is preserved.
2. Local Directional Descent Potential (${\Phi }_k$): The ability of a direction to support large, productive steps without exploding.

The Secret of Muon/Freon: They intentionally sacrifice alignment ($\gamma$)—ignoring the exact direction the gradient points—to maximize descent potential ($\Phi$). By flattening the spectrum, they allow for a large, constant step size that doesn't oscillate, which is far more important for deep learning than following the "exact" local geometry.

Conclusion & Future Outlook

The paper concludes that spectral optimizers are essentially stability-first rather than geometry-first. Muon's success isn't because it "finds the right shape," but because it ensures step-size optimality and suppresses noisy, large singular values.

Takeaway for Practitioners: Don't get bogged down in the "correct" geometric derivation. The most effective optimizers of the future will likely be those that effectively manage the Signal-to-Noise Ratio (SNR) in the mid-range of the singular value spectrum, potentially using even simpler stochastic redistributions than we use today.