Diffusion Consistency: Why Disjoint Training Sets Converge to the Same Image

TL;DR

Why do two diffusion models, trained on completely different subsets of images, generate the exact same face when given the same noise seed? This paper traces this "magic" to the spectral geometry of data. Using Random Matrix Theory (RMT), the authors show that diffusion models primarily learn to invert Gaussian noise based on shared covariance statistics, acting as a "renormalized" linear predictor that remains stable even when the specific training samples differ.

The Mystery of the Identical Twin Outputs

In the world of GANs or VAEs, the latent space is arbitrary; a rotation of the latent code usually results in a completely different image. Diffusion models are different. As shown in the motivating experiments, models like UNet and DiT, when trained on non-overlapping splits of the FFHQ dataset, map the same noise seed to strikingly similar outputs.

The authors' central insight is that this consistency isn't just a byproduct of "good training"—it's a linear effect. A simple Gaussian linear predictor (a Wiener filter) already explains most of the cross-split agreement.

Methodology: The RMT Lens

To formalize this, the authors move into the regime of Random Matrix Theory. They treat the empirical covariance matrix $\hat{\Sigma }$ as a random variable and ask: how does its randomness affect the learned denoiser $D_{\hat{\Sigma }}$?

1. Renormalized Noise Scale

The theory introduces a self-consistent relation where the raw noise ${\sigma }^2$ is transformed into a renormalized noise scale $\kappa ({\sigma }^2)$.

• Physical Intuition: When data is finite, the model "sees" less signal in low-variance directions. To compensate, it treats these directions as having more noise than they actually do, leading to "overshrinkage" toward the dataset mean.

2. The Variance Law

By extending "Deterministic Equivalence" to fractional matrix powers, the authors derived a beautiful factorization of the variance (the disagreement) between models: $extVar\propto (extAnisotropy)imes(extInhomogeneity)imes(extGlobalScaling)$

• Anisotropy: Disagreement is highest in directions where the data variance matches the noise level.
• Inhomogeneity: Samples displaced along high-variance modes (e.g., major facial features) show more uncertainty than those in low-variance modes.

Experimental Validation: From Linear to Deep

Does this linear theory hold for complex neural networks? The authors tested UNet and DiT models across resolutions (32px to 64px) and found a clear two-phase transition:

1. Memorization Phase (Small $n$): Models ignore the linear structure and simply memorize training points. Consistency is low.
2. Renormalization Phase (Large $n$): As $n$ grows (typically $n\ge 3000$), models converge toward the linear RMT prediction.

The spectral analysis reveals a harsh truth: High-frequency details (low-variance modes) take significantly more data to become consistent. While the "global shape" of a face stabilizes quickly, the specific textures require massive datasets before different models "agree" on them.

Deep Insight: Why Diffusion?

The paper concludes that diffusion models are unique because they learn a score vector field $abla\log p(x;\sigma )$. At high noise levels, the data distribution looks Gaussian, making the score field inherently stable and tied to the global covariance of the dataset. This "anchors" the generation process, providing a universal map from noise to structure that transcends the specific training architecture or data subset.

Conclusion & Limitations

While this work provides a rigorous baseline for reproducibility, it acknowledges that linear theory cannot capture architecture-specific inductive biases (like the "mosaic" effect of CNNs). However, the RMT framework provides a vital map of the "spectral uncertainty" in generative modeling, showing that our noise seeds are not just random—they are shaped by the very geometry of the data we train on.