The Ghost in the Noise: Why Independent Diffusion Models Dream the Same Dreams

TL;DR

Why do two diffusion models, trained on completely different sets of images, generate the exact same face when given the same random noise? This paper reveals that the answer lies in the spectral geometry of data. By applying Random Matrix Theory (RMT), the authors show that diffusion models are anchored by shared Gaussian statistics, behaving like a physical system where finite data "renormalizes" the noise level.

The Mystery of Generative Consistency

In the world of GANs and VAEs, the latent space is a hall of mirrors—arbitrary rotations mean that the same "seed" produces wildly different results across training runs. Diffusion models are different. They utilize a deterministic probability flow that seems to map noise to data in a remarkably stable way.

The authors noticed that this consistency isn't just a byproduct of deep learning magic; it's present even in linear Gaussian predictors. As shown in the motivation experiment, samples from disjoint splits are visually closer to each other than to their actual training neighbors, effectively ruling out simple memorization as the cause of similarity.

The Physics of Finite Data: Noise Renormalization

The core technical contribution is the use of Deterministic Equivalence (DE). In high dimensions, random empirical matrices (from finite data) can be replaced by deterministic surrogates.

1. The Renormalized Scale

When training with limited data, the model doesn't just see the true noise ${\sigma }^2$; it sees a "renormalized" noise level $\kappa ({\sigma }^2)$. The authors use the Silverstein equation to show that $\kappa ({\sigma }^2)>{\sigma }^2$.

• Physical Intuition: Finite data acts as an additional ridge penalty. This causes the model to "overshrink" low-variance directions, treating subtle signals as noise and pulling them toward the dataset mean.

2. The Variance Law

Why do models disagree on some parts of an image but not others? The authors derive a variance formula that factorizes into three parts:

• Anisotropy: Disagreement is highest in directions where the data variance matches the noise level.
• Inhomogeneity: If a noise seed is far from the mean along a high-variance axis, there is more room for the splits to disagree.
• Scaling: Disagreement vanishes at a rate of $1/n$ as the dataset grows.

From Linear Theory to Deep Networks

The most impressive feat of this paper is validating these RMT results on state-of-the-art architectures like UNet and DiT.

The Generalization Transition

The authors identify two distinct phases in diffusion training:

1. Memorization Phase ($n\le 1000$): The model simply reconstructs training samples. RMT does not apply here.
2. Renormalization Phase ($n\ge 3000$): The model begins to generalize. Here, the generated images strongly resemble the RMT-predicted linear solution.

As the dataset size grows, the deep networks converge toward the population linear predictor, confirming that the "ground truth" of these models is the underlying statistical structure of the data manifold.

Critical Insights & Future Work

This work provides a principled baseline for reproducibility. It tells us that if two models aren't producing consistent results, it’s likely because their training sets don't share identical second-order moments.

Limitations:

• The linear surrogate naturally underestimates the variability of highly non-linear, expressive models.
• It does not yet account for specific architectural inductive biases (like the locality of CNNs vs. the global reach of Transformers).

The Bottom Line: Diffusion models are consistent because they are experts at extracting the stable spectral geometry of a distribution. The "noise" we give them is not just random—it is a coordinates system anchored by the covariance of the data itself.