[Theoretical Breakthrough] Adam Converges: Solving the 7-Year Mystery of AI's Workhorse Optimizer

TL;DR

Since 2018, the AI community has lived with a paradox: the most popular optimizer, Adam, was mathematically proven to "diverge" by the ICLR Best Paper award winners Reddi et al., yet it continued to power every major LLM breakthrough from GPT-3 to DeepSeek. This paper finally resolves the conflict. It proves that Adam doesn't need a "fix"—it just needs the right hyperparameters. By identifying a phase transition in the $({\beta }_1,{\beta }_2)$ plane, the authors provide the first rigorous guarantee that vanilla Adam converges when ${\beta }_2$ is sufficiently large.

The "Divergence" Ghost

The celebrated counter-example by Reddi et al. [2018] showed that for any $({\beta }_1,{\beta }_2)$, one could construct a problem where Adam fails. This led to a flurry of "corrected" versions like AMSGrad and AdaBound.

The Insight: The authors of this paper noticed a subtle trick in that proof—the bad example was picked after fixing the hyperparameters. In reality, we do the opposite: we have a dataset (the problem) and then we find the best $({\beta }_1,{\beta }_2)$.

Figure 1: (a) The theoretical failure region from 2018 vs. (b) the actual convergence/divergence phase transition discovered in this work. Note the blue "safe" region.

Why Large ${\beta }_2$ is the Secret Sauce

The core mathematical challenge in proving Adam's convergence is the denominator $\sqrt{v_k}$. Because $v_k$ is a moving average of squared gradients, it can be extremely noisy or nearly zero, causing the update step to explode.

1. The Concentration Effect

The authors prove that when ${\beta }_{2}o1$, the term $1/\sqrt{v_k}$ concentrates around its expected value. Physical Intuition: A large ${\beta }_2$ acts like a heavy flywheel. It prevents $v_k$ from changing too rapidly, effectively "freezing" the adaptive part of the learning rate so it behaves predictably.

2. The Potential Function

To handle the momentum term $m_k$, they introduce an auxiliary sequence $z_k$: $z_k=\frac{x_k-{\beta }_1^n{x}_{k-n}}{1-{\beta }_1^n}$ This math allows them to "cancel out" the historical signals in Adam's memory, proving that on average, Adam still moves in a direction that decreases the loss function (the descent direction).

Experimental Validation: From MNIST to LLMs

The paper proves that the boundary for a "safe" ${\beta }_2$ is problem-dependent and specifically depends on the number of mini-batches ($n$).

• Decreasing Batch Size = Increasing $n$
• Finding: Smaller batch sizes require a larger ${\beta }_2$ to remain in the "blue" convergence zone.

Figure 2: Empirical evidence on MNIST showing the "Danger Region" where gradients diverge when ${\beta }_2$ is too small.

This theoretically explains why LLM practitioners (who often use small per-device batch sizes) have gravitated toward ${\beta }_2$ values of 0.95 or 0.999. The paper's suggestions are echoed in the technical reports of models like Llama and DeepSeek, where tuning ${\beta }_2$ inversely with batch size is a standard "trick" for stability.

Impact & Critical Analysis

This is a "Test-of-Time" caliber paper that bridges the gap between optimization theory and deep learning practice.

• Major Achievement: It removes the "Bounded Gradient" assumption. Most previous proofs assumed gradients couldn't grow to infinity—but a proof of convergence is useless if it assumes the thing it’s trying to prove!
• Limitation: The current convergence bound $O(\log T/\sqrt{T})$ includes constants of order $n^5$. While mathematically sound, it suggests the convergence might be slow in extreme cases, and these constants aren't yet "tight."

Summary for Practitioners

If your model training is unstable:

1. Don't panic and switch optimizers immediately.
2. Check your ${\beta }_2$: If you are using small batches, move ${\beta }_2$ closer to 0.99 or 0.999.
3. The Rule of Thumb: Keep ${\beta }_1<\sqrt{{\beta }_2}$. For the standard ${\beta }_1=0.9$, any ${\beta }_2>0.81$ is theoretically safe, provided it's at least above the batch-size threshold.

The discovery of a phase transition in the $({\beta }_1,{\beta }_2)$ plane provides a new map for navigating the complex landscape of neural network optimization.