From Score Approximation to Distribution Approximation: A Rigorous Theoretical Foundation for Score-Based Diffusion Models

TL;DR

This paper proves that if a neural network approximates the true score function of a diffusion process arbitrarily well in the L2 sense, then the probability distribution generated by the reverse diffusion process is arbitrarily close to the target data distribution in Kullback–Leibler divergence (up to an irreducible prior mismatch). The proof combines Hornik's universal approximation theorem, Girsanov's theorem on path space, and the data processing inequality, yielding the explicit bound: KL(μ || μθ) < δ + ½ g_max² ε. This is the first approximation-theoretic result that directly links the universal approximation capability of neural networks to the distribution approximation capability of score-based diffusion models.

Background and Positioning

Diffusion models have achieved state-of-the-art performance in image, audio, and molecular generation, yet their theoretical foundations remain incomplete. While classical universal approximation theorems guarantee that neural networks can approximate any function (including score functions), it was unclear whether this functional approximation guarantees distribution approximation after the reverse diffusion process. This work fills that gap by providing a rigorous, quantitative connection. It complements the statistical learning perspective of Chen et al. (2023) by focusing on the approximation-theoretic question: what is the minimal possible distribution error if we can approximate the score arbitrarily well (ignoring finite-sample noise)?

Problem & Motivation

The Core Puzzle:
Score-based diffusion models learn a neural network sθ(x,t) to approximate the true score function s(x,t) = ∇_x log p_t(x). The reverse SDE uses this learned score to generate samples. But even if sθ is a perfect approximation of s in function space, does that guarantee that the generated distribution μθ is close to the true data distribution μ? The answer is not obvious because the reverse SDE is a stochastic process, and the score affects the entire path measure.

Why Prior Work Falls Short:

• Universal approximation theorems (Cybenko, Hornik) only guarantee function approximation, not distribution approximation.
• Existing distribution guarantees (e.g., Chen et al. 2023) rely on structural assumptions (low-dimensional linear subspaces) and finite-sample statistics, leaving the pure approximation-theoretic question unanswered.
• Without a rigorous link, practitioners cannot be sure that improving score approximation (e.g., by increasing network capacity) will necessarily improve generated sample quality.

Author's Insight:
The key insight is to use Girsanov's theorem to compute the exact KL divergence between the path measures of the true and learned reverse SDEs. This expression directly involves the L2 score error integrated over time. Then, the data processing inequality tells us that the KL divergence between the marginal distributions at time 0 is bounded by the path-space KL divergence. This two-step decomposition cleanly separates the prior mismatch term from the score approximation term, leading to the final bound.

Methodology: The Core Theoretical Framework

Step 1: Score Approximation → Path Measure Approximation (Proposition 1)

The authors consider the true reverse SDE (with score s) and the learned reverse SDE (with score sθ), both starting from the same terminal distribution pT (or p_prior). By applying Girsanov's theorem, they derive an exact identity:

This is a remarkable result: the KL divergence between the entire path measures (P and Qθ) is exactly the sum of two terms:

1. The prior mismatch at the terminal time: KL(pT || p_prior)
2. The integrated L2 score error weighted by the diffusion coefficient g²(t)

No other sources of error appear. The proof uses the fact that both processes are driven by the same Brownian motion and that the drift difference is exactly g²(t)(sθ - s).

Step 2: Path Measure Approximation → Distribution Approximation (Proposition 2)

The data processing inequality for relative entropy states that for any two probability measures P and Qθ on a path space, the KL divergence between their marginal distributions at time 0 is bounded by the KL divergence between the full path measures:

This is a direct consequence of the contraction of relative entropy under the projection map (X_t)_{t∈[0,T]} → X_0. The proof uses the Radon–Nikodym derivative identity and Jensen's inequality.

Step 3: Combining with Hornik's Theorem (Theorem 1)

Finally, Hornik's universal approximation theorem guarantees that for any ε > 0, there exists a feedforward neural network sθ (with e.g., softplus activation) such that:

||sθ - s||_{L2(μ)²}² < ε

Putting everything together, we obtain the main result:

where δ = KL(pT || p_prior) and g_max = max_{t∈[0,T]} |g(t)|.

Key properties of the bound:

• Additive: The prior mismatch and score error contribute independently.
• Explicit: All constants are known (g_max, δ, ε).
• Dimensionality-free: The bound does not depend on the data dimension d (except through the score error itself, which is defined in L2(μ)²).
• Tight: In the ideal case where pT = p_prior (δ=0), the bound reduces to ½ g_max² ε, showing that score approximation directly controls distribution approximation.

Theoretical Results and Implications

Interpretation of the Bound

The theorem shows that the distribution approximation error is bounded by two independent sources:

1. Prior mismatch (δ): The inevitable error from using a simple prior (e.g., standard Gaussian) to approximate the true terminal distribution pT. This term can be reduced by designing better noise schedules (e.g., VP-SDE, sub-VP) that make pT close to Gaussian.
2. Score approximation error (ε): The L2 error of the neural network. This can be made arbitrarily small by increasing network capacity and training time (in the approximation-theoretic sense, ignoring statistical estimation error).

Comparison with Chen et al. (2023)

Chen et al. studied the finite-sample setting where the score is estimated from data, and they derived bounds under low-dimensional linear subspace assumptions. Their results involve sample complexity and dimension-dependent rates. In contrast, this paper focuses on the approximation error (the best possible error given perfect optimization and infinite data). The two results are complementary: one tells us how well diffusion models can approximate in principle (this work), and the other tells us how many samples are needed to achieve that approximation in practice (Chen et al.).

Practical Implications

• Noise schedule design: The bound highlights the importance of minimizing KL(pT || p_prior). This justifies the common practice of using VP-SDEs with large T so that pT is nearly Gaussian.
• Network capacity: Since the score error ε can be made arbitrarily small, the bound suggests that there is no fundamental limitation on the expressive power of diffusion models. The only irreducible error is the prior mismatch.
• Evaluation metric: The bound uses KL divergence, which is a strong metric (controlling many other divergences via Pinsker's inequality). However, KL is not symmetric and may be infinite if the distributions have disjoint supports; the paper's assumptions (e.g., Lipschitz score) ensure that the supports overlap.

Critical Analysis & Conclusion

Strengths

• Rigorous and elegant: The proof is concise, cleanly combining three classical results (Hornik, Girsanov, data processing inequality).
• General: The assumptions are mild (Lipschitz score, Novikov condition) and are satisfied by many practical architectures and noise schedules.
• Actionable: The bound provides clear guidance for practitioners: reduce the prior mismatch and improve score approximation.

Limitations

• KL divergence only: The result is specific to KL divergence. Other metrics (e.g., Wasserstein, total variation) are not covered, though bounds via Pinsker's inequality (TV ≤ √(½ KL)) extend the result to total variation.
• Approximation-theoretic, not statistical: The result assumes we can find a neural network that achieves an arbitrarily small L2 score error. In practice, finite data, optimization, and generalization errors add additional terms. The paper does not address sample complexity.
• Lipschitz assumption: The score function is assumed to be globally Lipschitz. This holds for Gaussian mixtures or strongly log-concave distributions, but may fail for heavy-tailed or multimodal distributions with sharp boundaries.

Future Directions

• Quantitative approximation rates: Hornik's theorem does not provide rates. Combining the framework with modern approximation theory (e.g., Barron space, ReLU networks) could yield explicit convergence rates in terms of network size.
• Alternative metrics: Extending the analysis to Wasserstein distance or Jensen–Shannon divergence would provide stronger geometric guarantees.
• Discrete-time and probability flow ODEs: The continuous-time SDE framework is the most elegant, but practical DDPMs use discrete steps. An analogous theorem for discrete-time processes would be valuable.
• Geometric and structured data: Extending to manifolds, graphs, or Riemannian diffusion models is an important open problem.

Takeaway

This paper provides the missing theoretical link between score approximation and distribution approximation in diffusion models. It shows that the universal approximation power of neural networks directly translates into universal approximation power of the generated distribution, up to an irreducible prior mismatch. The clean bound KL(μ || μθ) < δ + ½ g_max² ε serves as a theoretical foundation for the empirical success of score-based generative modeling and offers a clear principle for designing better diffusion models.