MD Decoupling: Killing Warmup and Weight Decay through Polar Optimization

TL;DR

Neural network training has long been plagued by a hidden "interference" between the size (magnitude) of weights and the way they rotate (direction). Magnitude–Direction (MD) Decoupling solves this by explicitly separating these two in the optimizer. The result? 2x compute efficiency on large MoEs, perfect learning rate transfer across model sizes, and the total removal of weight decay and warmup.

The Problem: The Magnitude-Direction Interference

In traditional optimizers like Adam, every weight matrix $W$ is updated as a single blob. However, $W$ is composed of a magnitude $\Vert W\Vert$ and a direction $\hat{W}=W/\Vert W\Vert$.

Experimental evidence shows that stepping in $W$ as a whole leads to a "coupling" problem:

1. Inverse Angular Rate: The angular change (how much the model "learns") is inversely proportional to the current magnitude. If a layer grows too large, learning stalls; if it's too small, units fly off in random directions.
2. Magnitude Drift: Even when the loss only cares about direction (common with LayerNorm/RMSNorm), updates tend to be perpendicular to the weights, which mathematically forces the magnitude to drift upwards.

To fix this, we've historically used Weight Decay and Warmup—essentially "patches" to stop weights from exploding or rotating too violently at the start.

Methodology: High-Fidelity Factorization

The researchers propose a simple but profound modification to the optimizer step. Instead of updating $W$, they factorize it: $W=\mathrm{diag}({\gamma }_{\mathrm{row}})\hat{W}\mathrm{diag}({\gamma }_{\mathrm{col}})$

Here, $\hat{W}$ is constrained to a fixed-norm Frobenius hypersphere. The "Gains" ($\gamma$) are learnable vectors that scale each row and column.

Why this works:

• Natural LR Control: The Learning Rate (LR) now directly controls the angular update on the sphere. No more drift, no more "stiff" weights.
• Fine-Grained Scales: By learning per-row and per-column gains, the model maintains the ability to amplify important features—something lost in simpler "spherical-only" training methods like nGPT.
• Fused Implementation: Crucially, this happens inside the optimizer. The model still sees one standard weight matrix, meaning no overhead in the forward or backward pass.

Experiments: Scaling to MoE and Beyond

The authors validated this on dense models up to 1.3B and Mixture-of-Experts (MoE) models up to 6.7B parameters.

1. The Death of Warmup

Dropping warmup in standard Adam usually crashes training. With MD Decoupling, dropping warmup actually improves the final loss because the model can start learning at full speed from step one without instability.

2. Zero-Shot LR Transfer

One of the "Holy Grails" of ML is picking an LR on a small model and having it work on a huge one. Because the relative update is fixed by the sphere, the optimal LR stays flat across model width and depth.

3. Compute Savings

On the Apertus 1.0 dataset mixture, Muon + MD Decoupling reached the same loss as a well-tuned AdamW baseline with half the compute.

Critical Analysis & Conclusion

This paper settles a long-standing debate in technical circles: Does the normalization axis matter? The authors find that while Frobenius, row-wise, or column-wise normalization all work, the gain is what matters. Adding per-row/column learnable gains is the secret sauce that makes spherical training competitive with (and superior to) the best-tuned AdamW recipes.

Takeaways

• Weight Decay is a Band-Aid: We only need it because our optimizers are geometrically poor. MD Decoupling replaces it with a geometric constraint.
• Architecture Agnostic: The logic lives in the optimizer, making it a "drop-in" upgrade for existing Transformer stacks.
• Future Scope: How does this impact 8-bit or 4-bit training? Given the stability of the magnitudes, MD Decoupling may be the key to ultra-low-precision LLMs.

Senior Editor's Note: This work is part of a broader shift toward "Manifold Optimization." By treating weights as vectors on spheres rather than points in Euclidean space, we are finally aligning our optimization mathematics with the actual training dynamics of deep networks.