Reviving ConvNeXt: Is Convolution the Secret to Efficient Diffusion Scaling?

TL;DR

As the AI community pivots heavily toward Diffusion Transformers (DiT), a new research paper titled "Reviving ConvNeXt for Efficient Convolutional Diffusion Models" challenges the status quo. The authors introduce FCDM (Fully Convolutional Diffusion Model), a backbone that uses modern ConvNet design to outperform DiT-XL/2 while using 50% fewer FLOPs and training 7x faster. It turns out, that the locality bias and efficiency of convolutions might be exactly what diffusion models need to scale without breaking the bank.

The "Transformer-Only" Delusion?

The current generative AI landscape is obsessed with Transformers. From Stable Diffusion 3 to Sora, the assumption is that the fully attentional architecture is the only way to achieve high-quality scalability. However, this comes at a massive cost:

• Quadratic Complexity: Self-attention becomes prohibitively expensive as resolution grows.
• Resource Hunger: Training state-of-the-art DiT models requires vast GPU clusters.
• Ignoring Priors: Transformers treat images as simple sequences of patches, often ignoring the natural spatial locality that convolutions are "born" to handle.

Methodology: High-Performance ConvNeXt for Denoising

FCDM isn't your grandfather's ResNet. It is built on the ConvNeXt philosophy—a "modernized" ConvNet that borrows the best spirits of Transformers (like large kernels and inverted bottlenecks) while staying purely convolutional.

Key Architectural Shifts:

1. Adaptive LayerNorm (AdaLN): The authors replaced standard LayerNorm with AdaLN to inject time and class embeddings, a trick pioneered by DiT but adapted here for spatial feature maps.
2. Inverted Bottleneck & Large Kernels: By using 7x7 depthwise convolutions and expanding channels within the block, FCDM achieves a wider receptive field and richer feature representation.
3. Global Response Normalization (GRN): Instead of heavy Attention layers, FCDM uses GRN to reduce channel redundancy, ensuring that the model doesn't waste capacity on overlapping features.

Figure: The FCDM block (left) and the scalable U-shaped architecture (right).

Performance: Efficiency That Defies the Trend

The most striking result is the Training Efficiency. While DiT-XL/2 requires roughly 7 million steps to reach peak performance, FCDM-XL hits superior FID scores in just 1 million steps.

Beyond just training speed, the Inference Throughput is nearly 3.4x faster than DiT at the same parameter scale. This makes FCDM a formidable candidate for on-device generative AI where latency and power consumption are critical.

Table: Quantitative comparison showing FCDM outperforming DiT across all model sizes.

Critical Insight: Why Does It Work?

The secret sauce lies in the Locality Bias. Diffusion modeling is essentially a pixel-level reconstruction task. Convolutions naturally focus on local textures and edges, which are essential for high-fidelity denoising. By using Large Kernels (7x7), FCDM bridges the gap between local processing and the global context offered by Transformers, without the overhead of the KV-cache or quadratic attention matrices.

Furthermore, frequency-domain analysis shows that FCDM preserves high-frequency components (sharp edges and fine details) better than DiT, which often produces "smoother" but less detailed noise predictions.

Conclusion & Future Outlook

FCDM proves that the industry's rush toward Transformers might have left a lot of efficiency on the table. For researchers and developers working on Video Diffusion or Medical Imaging, where 3D/4D data makes Transformers exponentially expensive, FCDM offers a blueprint for a more sustainable scaling path.

The Takeaway: Convolution isn't dead; it was just waiting for a modernization. FCDM is a clear signal that the "Scaling Laws" of generative models are not exclusive to Transformers.