Unified Landscapes of Optimization: From Local Descent to Global Acceleration
Lectures on optimization
This comprehensive lecture series synthesizes the mathematical foundations of convex optimization, from classical gradient descent to modern variance reduction and interior-point methods. It rigorous explores the query complexity of first-order oracles and establishes the theoretical limits of acceleration in various geometric settings.
TL;DR
Optimization is the engine of modern AI, yet its theoretical boundaries are often misunderstood. This synthesis explores the "Lectures on Optimization" by Sinho Chewi, mapping out the transition from simple Gradient Descent (GD) to accelerated flows, non-smooth subgradient methods, and the sophisticated geometry of Mirror Descent. It proves why acceleration works through the lens of continuous-time physics and how variance reduction tames the noise in Stochastic Gradient Descent (SGD).
The "Why" behind the "What": The Oracle Model
The fundamental limit of any optimization algorithm is defined by its Oracle Complexity. If we treat a function as a black box, Theorem 1.1 reminds us of a brutal reality: without structural assumptions like convexity, finding an approximate minimizer requires queries that grow exponentially with dimension .
The breakthrough insight is that convexity allows local information (gradients) to yield global consequences. However, even within the convex realm, not all algorithms are created equal.
Methodology: The Physics of Acceleration
Why do Nesterov’s Accelerated Gradient Descent (AGD) and Conjugate Gradient (CG) outperform standard GD? The notes demystify this via Accelerated Gradient Flow (AGF).
Standard GD is a first-order discretization of a dissipative system. AGD, however, is a discretization of a second-order ODE:
abla f(x_t) = 0$$ This represents a particle with mass and momentum. By adding a "friction" coefficient ($\gamma = 3/t$), we allow the system to conserve energy enough to "overshoot" safely and converge at an $O(1/t^2)$ rate, effectively "squaring" the efficiency of standard GD.  *Figure 1: Comparison between monotonic Gradient Flow and the oscillatory yet faster Accelerated Flow.* ## Mirror Descent: Optimization on Manifolds One of the most profound shifts in modern optimization is moving beyond the Euclidean norm. If your parameters live on a probability simplex (like weights in a Transformer or a portfolio), the Euclidean distance is a poor metric. **Mirror Descent** solves this by mapping the primal space to a dual space using a **Mirror Map** $\phi$. The key innovation here is the **Bregman Divergence** $D_\phi(x, y)$, which replaces $|x-y\|^2$. For the simplex, using the negative entropy as a mirror map leads to the **Multiplicative Weights Update** algorithm: $$x_{n+1} \propto x_n \odot \exp(-h abla f(x_n))$$ As shown in Section 10, this "geometry-aware" approach reduces the complexity overhead from $O(d)$ to $O(\log d)$, a critical advantage in high-dimensional feature spaces. ## Stochasticity and Variance Reduction In the era of Big Data, we rarely compute full gradients. We use SGD. While SGD is computationally cheap, its noise prevents it from reaching high precision quickly. The notes provide a deep dive into **SVRG (Stochastic Variance Reduced Gradient)**. By periodically computing a "full" anchor gradient and using it to center local estimates: $$\hat{ abla}_n^t f = abla f_{i_n}(x_n) - abla f_{i_n}(\bar{x}_0) + abla f(\bar{x}_0)$$ SVRG ensures that as the algorithm approaches the minimizer, the variance of the updates tends to zero. This allows it to achieve **linear convergence** $(O(\log 1/\epsilon))$ on finite sums, matching the behavior of deterministic GD but at SGD prices.  *Table 1: The hierarchy of optimization rates, showcasing the dominance of ASVRG in finite-sum settings.* ## Critical Analysis & Conclusion ### Takeaway The theoretical framework provided here proves that optimization is not just about "going downhill." It is about understanding the **intrinsic geometry** of the loss surface and the **physical dynamics** of the algorithm. ### Limitations A major caveat is the reliance on **self-concordance** for second-order methods and the assumption of **unbiased oracles** for SGD. In deep learning, the "finite sum" structure is often violated by data augmentation, and the high-order derivatives required for Interior Point methods are often too costly to compute. ### Future Work The next frontier lies in **hyper-parameter hedging** (like the Silver Step size schedule) and bridging the gap between spectral properties of matrices and the generalization capabilities of ASGD in non-convex regimes.