[Theoretical Insight] How ReLU Activation Shapes Implicit Bias in High-Dimensional Neural Regression

TL;DR

Why does Gradient Descent (GD) find solutions that generalize well even without explicit regularization? This paper tackles this "Implicit Bias" mystery for one-hidden-layer ReLU networks. By moving into the high-dimensional regime ($d\gg n$), the authors prove that GD-trained ReLU models converge to solutions remarkably close—but not exactly equal—to the minimum-ℓ2-norm solution, with a quantifiable proximity gap of $\Theta (\sqrt{n/d})$.

Positioning the Work: Beyond the Worst-Case

In the world of linear regression, GD famously converges to the minimum-norm interpolant. But for ReLU networks, the landscape is a mess of non-convexity. Previous research hit two extremes:

1. The Pessimistic View: Vardi & Shamir (2021) showed that in the worst case, implicit bias doesn't even exist—GD can go anywhere.
2. The Idealized View: If data is perfectly orthogonal, GD finds the minimum-norm solution (Boursier et al., 2022).

This paper provides the "Missing Link": Real-world high-dimensional data is near-orthogonal. By leveraging high-dimensional concentration, the authors show that the non-convexity of ReLU essentially "freezes" into a manageable state.

The "Primal-Dual" Intuition

The researchers didn't just track the weights ($w$). They introduced a primal-dual representation:

• Primal ($\beta$): The model's predictions on the training data.
• Dual ($\alpha$): The coefficients representing weights in the span of the data.

In a ReLU network, the sign of $\beta$ determines if a neuron is "active." The paper's mathematical breakthrough involves showing that in high dimensions, the Activation Pattern stabilizes almost immediately.

Visualizing the Transition

The logic is captured in a transition diagram. Once an example's prediction aligns with its label sign, it stays active (Primal stability). Conversely, if a dual variable becomes sufficiently negative, the ReLU neuron "turns off" for that example and stays off (Dual stability).

Methodology: The Geometry of Near-Orthogonality

Why does high dimensionality simplify things? The authors use Random Matrix Theory (specifically properties of the Gram Matrix $X{X}^{o}p$) to show that $X{X}^{o}p$ concentrates around $\Vert \lambda {\Vert }_{1}I$.

• Physical Intuition: In high dimensions, every data point is nearly perpendicular to every other point. This "weak interaction" means that when GD updates the weight to fit point A, it barely disturbs the fit for point B.

Key Results: The Proximity Bound

The paper's "crown jewel" is the characterization of the distance between the GD limit ($w^{\infty }$) and the minimum-norm solution ($w^{\star }$).

Theorem Preview: $extLowerBound\approx \frac{\sqrt{n_{-}{y}_{min}^2}}{\sqrt{\Vert \lambda {\Vert }_1}}\le \Vert w^{(\infty )}-w^{\star }{\Vert }_2\le \frac{\sqrt{16n_{-}{y}_{max}^2}}{\sqrt{\Vert \lambda {\Vert }_1}}\approx extUpperBound$

This tells us that as $do\infty$, the distance goes to zero. However, the solutions are not identical. The "memory" of the initialization and the specific path GD takes to suppress negative labels creates a persistent, albeit small, discrepancy.

Table 1: Contextualizing ReLU model results against linear baselines.

Critical Analysis: What’s the Catch?

While the results are elegant, two limitations remain:

1. Initialization Sensitivity: The proof relies on "small initialization" or specific "disjoint" patterns for many neurons ($m>2$). If you initialize with large weights, the activation pattern might jump around too much for this analysis to hold.
2. The "Moderate" Dimension Problem: When $d$ is only slightly larger than $n$, the "near-orthogonality" breaks down. Simulation results in the Appendix show that in these cases, GD does an "example selection" process that is highly dependent on initialization—making it much harder to predict the final bias.

Future Outlook

This work shifts the focus of Deep Learning theory from "Global Landscapes" to "Local Stability." It suggests that overparameterization isn't just about capacity; it's about creating a geometric environment where optimization becomes simpler and more predictable. The challenge now is extending this to deep networks where "feature learning" creates dependencies that might break the near-orthogonality we rely on here.