Rethinking Generalization: Why Diffusion Models Succeed by Failing to Converge

TL;DR

Current theory struggles to explain why diffusion models generate novel images instead of just copying their training data. This paper argues that the "lack of memorization" in practical models is simply a matter of scale: memorization takes a very long time, and we usually stop training before it happens. However, the path to that "failure" is where the magic (fidelity) happens, governed by an unexpected "double descent" in distribution space.

Background: The Generative Paradox

In supervised learning, "interpolation" (perfectly fitting the training data) is often a sign of a good model. In diffusion models, interpolation is a disaster. If a diffusion model perfectly learns the score function of the training set, it becomes a kernel density estimator that only samples exact replicas of the training images.

The central mystery is: if the global minimum of our loss function is a "memorization state," why do our models produce beautiful, novel images?

The Fallacy of the Train-Test Gap

The authors first debunk a common misconception: that generalization in generative models can be measured by comparing performance on training vs. testing data.

As shown above, traditional metrics like FID or Sliced Wasserstein distance behave almost identically for both the training set and the test set throughout the entire training process. Even when the model starts memorizing (producing copies), the distance to the test set doesn't necessarily diverge. This means generalization in diffusion is not about the gap between train and test, but about the emergence of novelty and fidelity before the model collapses into memorization.

Methodology: Tracking the Memorization Transition

The researchers trained U-Net architectures on CIFAR-10 while sweeping through four variables:

1. Dataset Size (N)
2. Model Size (P)
3. Batch Size (B)
4. Learning Rate ($\eta$)

They discovered a "normalized step" formula that collapses the curves, proving that memorization timing is highly predictable: $T=au\cdot \frac{N_{\min }}{N}\cdot \frac{B}{B_{\min }}\cdot \frac{\eta }{{\eta }_{\min }}$

Key Insights from the Empirical Sweep:

• The Linear Scaling Law: The time it takes to start memorizing ($a{u}_{mem}$) scales linearly with the number of training samples. For massive datasets like LAION, the model would likely need to train for an impossible amount of time to reach the memorization phase.
• The Double Descent Mystery: They observed a "double descent" in distributional distance (FID). During training, quality improves, then briefly gets worse, then improves again before eventually degrading into memorization. This suggests different signal components (coarse structure vs. fine detail) are learned at different speeds.

The Role of Optimization

Perhaps the most practical finding involves how hyperparameters shape the "Peak Fidelity" (the best quality achieved before memorization).

1. Lower Batch Sizes = Better Fidelity: Contrary to the idea that big batches are always better, smaller batches provide a type of implicit regularization that pushes the model toward higher-quality generation.
2. Higher Learning Rates = Better Fidelity: Training at the "Edge of Stability" (large $\eta$) actually prevents the model from falling into the sharp, "memorizing" minima of the loss landscape, keeping it in the broader, generalizing regions.

Critical Analysis & Future Outlook

The paper concludes that "Why don't they memorize?" is a solved question: scale and early stopping protect us. The real open questions are:

• Metric Problem: How do we mathematically distinguish a model that is "inspired" by a data manifold from one that is purely "copying" local patches?
• Trajectory Theory: We need a theory that describes what happens during training, not just what the model looks like at infinite time convergence.

Takeaway: If you want better images, don't just look for the lowest training loss. Instead, leverage large learning rates and careful early stopping to catch the model at its "peak fidelity" window before the inevitable drift toward memorization begins.

Reference: Marion, P., & Wu, Y. (2026). Understanding diffusion models requires rethinking (again) generalization. Position Paper.