Deeper or Wider? Resolving the Architecture Dilemma via Sobolev Generalization Theory

TL;DR

In the world of deep learning, we often ask: should we make our models deeper or wider? This paper provides a rigorous mathematical answer: it depends on your data budget and your loss function. By analyzing Sobolev losses—which penalize errors in both values and derivatives—the authors prove that deep models are superior when data is plentiful and the task requires high regularity (like solving PDEs), whereas wide models are safer bets when data is scarce.

The Core Conflict: Super-convergence vs. Sampling Complexity

The research landscape has long known two contrasting facts:

1. Deeper Neural Networks (DeNNs) can approximate complex functions with exponentially fewer parameters than shallow ones (a phenomenon called "super-convergence").
2. Wider Neural Networks (WeNNs) are often easier to train and exhibit more stable generalization behavior because their hypothesis space is less "erratic."

The authors identify a critical gap: most previous theories assumed network parameters were bounded by a small constant. This assumption "handicaps" deeper networks, preventing them from using the very complexity that makes them powerful. By utilizing pseudo-dimension and uniform covering numbers, this paper breaks those chains to find the true Optimal Generalization Error.

Methodology: The Sobolev Perspective

The authors focus on Sobolev training, where the loss function $H^k$ involves derivatives up to order $k$.

• $k=0$ (L2 Loss): Standard supervised learning.
• $k=1$ (H1 Loss): Used in the Deep Ritz method for solving variational PDEs.
• $k=2$ (H2 Loss): Used in Physics-Informed Neural Networks (PINNs) to minimize residuals.

1. The Super-convergence Advantage

The paper highlights that DeNNs (with $W$ parameters) achieve an approximation error of $O(W^{-4(n-k)/d})$, while WeNNs only reach $O(W^{-2(n-k)/d})$. This means that for a fixed parameter budget, DeNNs are fundamentally more expressive.

2. The Sampling Penalty

However, depth comes at a cost. The sampling error for DeNNs scales with $O(W^2/M)$, compared to $O(W/M)$ for WeNNs. As the network gets deeper, you need significantly more sample points ($M$) to "tame" the high-complexity space and prevent overfitting.

Table 1: Comparison of Approximation and Sampling Errors for WeNNs and DeNNs.

Phase Transitions in Architecture Selection

The most striking contribution of the paper is the "Phase Diagram" (Figure 1). It maps out the regions where DeNNs beat WeNNs and vice versa.

Figure 1: The transition curves for different Sobolev losses. As the derivative order $k$ increases, the "DeNN-preferred" region (to the left/top) expands.

Key Insights from the Diagram:

• Data-Restricted? If $M$ (samples) is small, go Wide.
• Parameter-Restricted? If you have a cap on $W$, go Deep.
• PDE Solving? Higher-order derivatives (H1, H2) favor deeper architectures because they better capture the "Regularity" of the target function.

Empirical Validation: The Theory in Action

The authors tested their theory by solving a 2D function approximation task.

• In the Large Data Regime (10,000 points), the Deep Network (Depth 4) significantly outperformed the Shallow Network.
• In the Small Data Regime (1,000 points), the Shallow Network achieved much lower error, as the Deep Network overfitted.

Table 3: Performance of shallow vs. deep networks across different data regimes.

Conclusion and Future Outlook

This work provides a rigorous framework for network design. It tells us that the "best" architecture is not a static property of the model but a dynamic result of the available resources.

Looking forward, the authors suggest extending this analysis to overparameterized regimes (where the number of parameters exceeds the number of samples) and modern architectures like CNNs. For anyone building PINNs or high-precision industrial AI models, this paper is a blueprint for making the "Deep vs. Wide" decision with mathematical confidence.

Takeaway for your next project:

If you are training a PINN to solve fluid dynamics ($H^2$ loss) and you have access to a large cluster to generate or collect data, don't be afraid to go deep. The higher regularity of your loss function actually makes deeper networks more stable and accurate than their wide counterparts in the long run.