Scale Space Diffusion

Scale Space Diffusion: Why Process Noise at Full Resolution?

Scale Space Diffusion

Summary

Problem

Method

Results

Takeaways

Abstract

The paper introduces Scale Space Diffusion (SSD), a novel generative framework that unifies scale-space theory with diffusion models by embedding image downsampling directly into the forward noising process. By utilizing a specialized Flexi-UNet architecture, the model achieves high-quality image synthesis (CelebA, ImageNet) with significantly reduced computational overhead, outperforming standard DDPMs in training and inference speeds.

TL;DR

Researchers from the University of Maryland have bridged a gap between classical Computer Vision (Scale-Space Theory) and modern Generative AI (Diffusion Models). They introduce Scale Space Diffusion (SSD) and Flexi-UNet, a system that shrinks the image as it gets noisier. The result? A model that trains 2x faster and uses nearly 60% less compute, without sacrificing the fine details of the final image.

The Intuition: Squinting at Noise

If you look at a diffusion process in reverse, the early stages (high noise) only determine the "blobs" and "coarse structures" of an image. If you squint at a noisy face, you can't see the pores or the eyelashes; you only see the silhouette.

The authors asked a fundamental question: Why are we using a massive 256x256 UNet to process a "blob" that only contains 8x8 pixels worth of actual information?

By mapping Diffusion Timesteps ( $t$ ) to Scale Space Resolutions ( $r$ ), they proved that as noise increases, the "Information Density" drops in a way that matches a Gaussian Pyramid.

Methodology: The Math of Shrinking

The core challenge of shrinking images during diffusion is that standard DDPM math assumes isotropic noise (noise that is independent and identical for every pixel). When you downsample or blur an image, you correlate the pixels, breaking the DDPM assumptions.

1. Generalized Linear Diffusion

The authors extended the forward diffusion equation to include a linear operator $M_{t}$ : $x_{t} = M_{t} x_{t - 1} + η_{t}$ Where $M_{t}$ is a resizing operator. Because $M_{t}$ correlates noise, the "Reverse Process" becomes much harder.

2. Solving the Non-Isotropic Posterior

To sample from the resulting non-isotropic Gaussian distribution during inference, the authors utilized the Lanczos algorithm. This allows the model to "calculate" the square root of a massive covariance matrix implicitly, ensuring the noise added during upsampling steps perfectly matches the mathematical requirements of the diffusion chain.

3. Flexi-UNet: The Dynamic Backbone

To actually get a speedup, you can't just resize the image and pass it through a full-sized UNet. Flexi-UNet allows the model to:

Bypass early layers for low-resolution inputs.
Utilize 1x1 Convolutions to enter the network at the appropriate depth.
Perform asymmetric upsampling to transition between resolutions (e.g., from 32x32 to 64x64) during the denoising process.

Figure: The Flexi-UNet architecture routes different resolutions through specific subsets of the network blocks, saving massive amounts of FLOPs.

Experiments: Faster, Cheaper, Better

The authors tested SSD on CelebA (faces) and ImageNet (diverse objects).

Performance Scaling

As the target resolution increases, the benefits of SSD become more dramatic. At 256x256 resolution, the SSD (6-level) model achieves parity in quality while being significantly faster than the standard ADM (Ablated Diffusion Model) baseline.

Method	Resolution	Training Time (hrs)	GFLOPs	FID (Lower is better)
DDPM (Baseline)	256	87.31	497.03	5.52
SSD (6-Level)	256	42.88	209.69	13.50*
Note: while FID increases at 6-levels, SSD (3-level) maintains a close 7.79 FID with substantial savings.

Visual Results Figure: The progression of "clean" image predictions over the SSD process, showing how the model builds up from an 8x8 coarse blob to a 256x256 high-fidelity face.

Critical Insights & Takeaways

Efficiency is Not Just for Latent Space: While models like Stable Diffusion use a Latent Space (LDM) to save compute, SSD shows that Pixel-Space models can be made equally efficient by respecting the information hierarchy.
Implicit Transpose Trick: The use of torch.autograd.grad to compute the transpose of an implicit resizing operator is a clever engineering trick that makes the complex math of non-isotropic diffusion practical.
Limitations: The "Resolution Schedule" $r (t)$ is a new hyperparameter. Choosing a schedule that shrinks the image too aggressively (ConvexDecay) can hurt the FID, requiring a careful balance between speed and quality.

Conclusion

Scale Space Diffusion is a mathematically grounded refinement of the diffusion process. By acknowledging that noise level and resolution are two sides of the same coin, it allows us to stop wasting GPU cycles on high-resolution noise and focus compute where the information actually lives.

Find Similar Papers