The Variational Origin of Edge of Stability: Why GD is Bound to the 2/η Threshold

TL;DR

The "Edge of Stability" (EoS) is a bizarre phenomenon where neural network training pushes the Hessian’s largest eigenvalue exactly to $2/\eta$. This paper provides the first global explanation for this behavior using a new mathematical tool called the edge coupling. It proves that as long as the loss decreases, the trajectory is mathematically "forced" to hit the $2/\eta$ curvature boundary, regardless of where it starts.

Background: The Mystery of the 2/η Ceiling

In classical optimization theory, we are taught that the learning rate $\eta$ must be less than $2/{\lambda }_{max}$ to ensure convergence. However, modern deep learning ignores this. Early in training, networks undergo "progressive sharpening" until they hit the Edge of Stability, where the loss starts to oscillate but continues to decrease on average.

The core question has always been: Why this specific number? Why does every architecture, from MLPs to ResNets, gravitate toward this specific instability?

The "Edge Coupling": A New Lens

The author, Elon Litman, reformulates Gradient Descent (GD) not as a sequence of steps, but as a critical condition of a functional called the edge coupling:

${\mathcal{A}}_{\eta }(x,y)riangleqL(x)+L(y)-\frac{1}{2\eta }\Vert x-y{\Vert }_2^2$

By setting the gradient of this functional with respect to $x$ to zero, you recover exactly the GD update $y=x-\eta ablaL(x)$. This simple yet powerful definition allows us to treat GD iterates as physical states governed by "discrete mechanics."

Figure 1: Sharpness saturates near the 2/η = 4 threshold while the loss oscillates. Note how effective curvature (blue) tracks the threshold more accurately than raw sharpness (green).

Methodology: Global Forcing through Conservation

The paper’s most significant contribution is moving from local analysis (what happens at the edge) to global forcing (why we go to the edge).

1. The Telescoping Loss Formula

By expanding the edge coupling, the author derives a precise formula for loss change: $L(w_{k+1})-L(w_k)=-\frac{\Vert d_k{\Vert }_2^2}{2\eta }(2-\eta ilde{r}_k)$ where $ilde{r}_k$ is the "effective curvature." When we sum this over the entire training trajectory, the loss side "telescope" (everything in the middle cancels out), leaving only the total loss drop.

2. The Forcing Logic

Since the total loss drop is finite, but the training continues (steps are taken), the term $(2/\eta -ilde{r}_k)$ must spend time near zero. If the curvature $ilde{r}_k$ stayed much smaller than $2/\eta$, the loss would drop to negative infinity—which is impossible. Therefore, the trajectory is physically forced to visit regions where the curvature is exactly $2/\eta$.

Experiments and Bifurcations

The paper doesn't just theorize; it analyzes Period-Two Orbits. At the Edge of Stability, the model essentially bounces between two points. This is a "Period-Doubling Bifurcation."

For two-layer linear networks, the author proves that this bifurcation is width-invariant. It doesn't matter how many hidden neurons you have; the "normal form" of the instability remains the same.

Figure 2: Validation of the Pitchfork diagram. As η crosses the critical threshold, the amplitude of the oscillation follows a specific square-root scaling, regardless of overparameterization.

Why the System Doesn't Explode

If we are at the "edge" of instability, why doesn't the gradient blow up? The author identifies two mechanisms:

1. Growth above the threshold: If curvature exceeds $2/\eta$, the step size grows geometrically, which quickly pushes the model back into a "flatter" region of the landscape.
2. Oscillatory Cancellation: Near the edge, the multiplier is roughly $-1$. The updates alternate signs, effectively "canceling out" potential drift and keeping the parameters trapped near the edge rather than letting them fly off into infinity.

Critical Insight & Conclusion

This paper fundamentally redefines the Edge of Stability. It is not an accident of neural network depth or specific datasets; it is a mathematical attractor of discrete gradient descent.

Key Takeaways:

• Universal Threshold: The $2/\eta$ limit is baked into the vary nature of the GD loss-change formula.
• Efficiency of Instability: EoS training is not "failed" optimization; it is the system navigating a narrow ridge of maximal allowable progress.
• The Power of Averaging: By using "step-averaged Hessians" instead of instantaneous ones, the theory matches empirical data with near-perfect precision.

While the paper focuses on $C^2$ and $C^4$ smooth losses (excluding standard ReLUs), it provides the most rigorous bedrock to date for understanding why our models train at the brink of chaos.