Why does NorMuon's adaptive scaling degenerate, and how does DGA-Muon fix it?

NorMuon scales rows of the orthogonalized momentum

Muon updates a two-dimensional weight matrix along the direction of the orthogonalized momentum rather than the raw momentum. Given momentum M_t, it forms Orth(M_t) = UV^T from the thin SVD M_t = UΣV^T, which sets every singular value to one and keeps only the directional structure; that matrix UV^T is the polar factor. In practice Muon approximates this with Newton–Schulz iterations instead of a full SVD.

NorMuon adds a second-moment estimate v_t in R^m that tracks the row-wise mean square of the orthogonalized momentum, updated as an exponential moving average with decay β2. The parameter update then divides the orthogonalized momentum by the square root of v_t plus a small constant δ, broadcast row-wise. The key design choice is the source of that signal: unlike AdaGrad, Adam, or Shampoo, NorMuon derives its scaling from the orthogonalized momentum, not from historical gradients, which tightly couples adaptivity to the orthogonalization step.

Exact orthogonalization collapses scaling to one scalar

For square and wide matrices (m ≤ n), the paper expresses the row-wise mean of the element-wise square of O_t through the Gram matrix: mean_cols(O_t ⊙ O_t) = (1/n) diag(O_t O_t^T). Assuming M_t has full row rank, the SVD gives U orthogonal and V column-orthonormal, so exact orthogonalization yields O_t O_t^T = U(V^T V)U^T = UU^T = I_m. The matrix is therefore row-orthonormal, and the row-wise mean square becomes (1/n)·1_m, identical across all rows.

Substituting this into the EMA recursion and unrolling it with v_0 = 0 gives v_t = (1 − β2^t)·(1/n)·1_m, which is uniform across rows and varies only through a global scalar over time. The paper notes that applying the standard bias correction NorMuon omits would cancel the time-dependent factor entirely, leaving exactly (1/n)·1_m. Either way, the scaling vector 1/√(v_t + δ) assigns the same coefficient to every row, so the intended neuron-wise adaptivity degenerates into a trivial global scalar.

The orthogonality residual is the only source of variation

Real implementations use a finite number of Newton–Schulz iterations, so the approximately orthogonalized matrix satisfies O_t O_t^T = I_m + E_t, where E_t is the orthogonality residual. The row-wise mean square then becomes (1/n)(1_m + diag(E_t)), and unrolling the EMA gives v_t = (1 − β2^t)(1/n)1_m + R_t, where R_t = ((1 − β2)/n) Σ_{i=1}^{t} β2^{t−i} diag(E_i). The first term is exactly the row-invariant term from the exact case; R_t is the only source of row-wise variation.

Assuming the residual is bounded in spectral norm by a small ε, the diagonal entries satisfy |(E_t)_jj| ≤ ε, which bounds ‖R_t‖_∞ ≤ ε/n. The difference between any two entries of v_t is therefore at most 2ε/n, and by the Mean Value Theorem the induced difference in the scaling factors 1/√(v_t + δ) is bounded by Cε/n for some constant C. Because Newton–Schulz drives ε very small in practice, the row-wise scaling factors vary only negligibly. The paper's interpretation is that NorMuon's apparent adaptivity is a byproduct of inexact orthogonalization rather than a response to gradient statistics or curvature.

Better orthogonalization makes NorMuon less adaptive

The analysis yields what the authors call the Orthogonalization–Adaptivity Paradox: NorMuon's adaptivity is inversely coupled to orthogonalization quality. More accurate orthogonalization weakens the row-wise variation; in the exact limit for square and wide matrices it disappears entirely. Less accurate orthogonalization permits larger variation and therefore stronger apparent adaptivity.

The paper reads this as evidence that NorMuon's adaptive mechanism compensates for imperfections in orthogonalization rather than supplying optimization-driven adaptivity. That places it in tension with a separate line of Muon research aimed at making orthogonalization more accurate or more efficient, since improvements on that axis would shrink the very residual NorMuon's scaling depends on. The analysis assumes full row rank; the paper treats rank deficiency as an undesirable edge case rather than the generic setting.

Tall matrices expose a geometry mismatch in row scaling

For tall matrices the picture differs. Orthogonalization enforces only one-sided orthogonality, namely column orthonormality, not row orthonormality. The total squared ℓ2 norm across all rows is fixed, but the squared ℓ2 norm of each individual row is unconstrained, so the row-wise scaling factors can vary even under exact orthogonalization. That variation comes from the non-uniform distribution of row energy after orthogonalization, not from gradient information.

The paper further argues that NorMuon's rigid row-wise scaling is geometrically misaligned with this one-sided structure. For a tall matrix the polar factor is column-orthogonal, with its columns defining the orthogonal directions, while row-wise scaling acts along the row dimension and distorts the column geometry, breaking the orthogonality the polar factor induced. This is a structural mismatch rather than a tuning problem.

Two design principles decouple and realign adaptivity

From these limitations the paper derives two principles for a desirable adaptive mechanism in Muon. First, adaptive scaling should be decoupled from orthogonalization, with scaling factors computed directly from raw gradients and then applied to the polar factor. Second, adaptive scaling should be aligned with the shape-dependent orthogonal structure of the polar factor: row-wise scaling for wide matrices and column-wise scaling for tall matrices, with either choice acceptable for square matrices. The paper provides a formal proof that this geometry-aligned scaling preserves the orthogonal structure of the update.

DGA-Muon adds three further design choices. The second-moment estimate is built from the sum rather than the mean of squared entries, so each row or column gradient vector is treated as one adaptive unit scaled by its total accumulated gradient energy; units with larger accumulated energy get smaller scaling factors. Standard bias correction is applied to the second-moment estimate, since it is computed from raw gradients and suffers the usual EMA initialization bias, while the first moment needs none because orthogonalization is scale-invariant. Finally, an adaptive clipping operation bounds the scaling factors between c·s_med and d·s_med around the median, limiting the condition number of the induced diagonal scaling matrix and preserving Muon's ability to support large step sizes.

NanoChat results separate the theory from the baseline

The paper reports two empirical checks. First, it tests the theoretical claim that NorMuon's scaling degenerates on square matrices by removing adaptive scaling on the Q/K/V/O projection matrices in NanoChat, where those matrices are optimized as full matrices over all attention heads and are therefore square. Training runs 7,308 steps with validation every 250 steps, using validation bits per byte (BPB), where lower is better and the metric is invariant to tokenizer vocabulary size. Removing the scaling leaves validation BPB essentially unchanged, with the two curves nearly overlapping, which the authors present as empirical support for the degeneration analysis.

Second, DGA-Muon is compared against NorMuon end to end on a 24-layer, 1.38B-parameter NanoChat Transformer with the data-to-parameter ratio raised from 10 to 18. DGA-Muon reaches a final validation BPB of 0.6910 versus 0.6924 for NorMuon, and a final training loss of 2.2708 versus 2.2799. On the DCLM CORE benchmark, DGA-Muon scores 0.3089 against NorMuon's 0.2942, an absolute improvement of 0.0147 and a relative gain of 5.0%. The paper notes that the two optimizers need different hyperparameters because their scaling signals differ, and that DGA-Muon uses a more gradual learning-rate schedule with extended warmup, a larger warmdown ratio, and a lower final learning-rate fraction.

Where the claim holds and where it stops

The degeneration result is stated for exact orthogonalization under a full row rank assumption, with square and wide matrices collapsing to a global scalar and tall matrices retaining variation from row-energy distribution. The residual-based bound on scaling variation is derived under a spectral-norm bound ε on the orthogonality residual, which the paper says matches practical Newton–Schulz tolerances. The empirical verification of degeneration is specific to the square Q/K/V/O projection matrices in NanoChat, and the end-to-end comparison uses one architecture, one data-to-parameter ratio, and separately tuned hyperparameters for each optimizer.

The paper also frames its contribution more broadly than a single optimizer fix: it argues that adaptive mechanisms for matrix-aware optimizers should separate the extraction of optimization-relevant signals from matrix transformations, and that row-wise or column-wise adaptivity is a natural response to the directional structure of matrices rather than merely an approximation to element-wise adaptation. The practical priority the evidence supports is the decoupling and alignment choice itself: when comparing adaptive Muon variants, the question to ask is whether the scaling signal comes from raw gradients and whether its axis matches the polar factor's one-sided orthogonal structure, since those two properties are what separate DGA-Muon's 0.3089 CORE score from NorMuon's 0.2942 in this evaluation.

Sources used in this answer