GAT: Resurrecting GANs through Pure Transformer Scaling

TL;DR

Generative Adversarial Transformers (GAT) successfully scale GANs up to XL sizes by combining a pure ViT-style architecture with the VAE latent space. By solving the twin problems of early-layer collapse and scale-dependent instability, GAT achieves SOTA single-step performance (FID 2.18 on ImageNet-256) with 4x higher training efficiency than traditional baselines.

The Scaling Paradox: Why GANs Remained "Small"

While the industry moved toward Transformer-based Diffusion (DiT) and Autoregressive (VAR) models that scale predictably with compute, GANs were left behind. Prior attempts at large-scale GANs often relied on "task-specific hacks" rather than genuine architectural scaling. The authors identify two fundamental blockers:

1. Layer Inactivity: In deep GANs, the early layers of the generator often become redundant, serving as near-identity mappings.
2. Training Speed Asymmetry: As models get wider, the functional update magnitude per step increases, causing the adversarial game to break (divergence).

Methodology: The GAT Blueprint

The GAT framework utilizes a pure Vision Transformer (ViT) backbone for both the Generator (G) and Discriminator (D), operating within the latent space of a pre-trained VAE.

1. Activating Early Layers (MNG)

To ensure every block earns its keep, authors introduced Multi-level Noise-perturbed image Guidance (MNG). The generator produces auxiliary outputs at different stages. Earlier stages are supervised by matching real data perturbed by heavy noise, while later stages handle clean data. This creates a "coarse-to-fine" learning trajectory across the model's depth.

2. The Width-Aware Learning Rate Rule

To stabilize training as models grow, authors proposed a simple yet effective scaling rule: ${\eta }_{\mathrm{adapt}}={\eta }_{base}\cdot \frac{C_{base}}{C_{model}}$ By decreasing the learning rate ($\eta$) inversely to the channel width ($C$), GAT maintains a consistent "functional update magnitude," preventing the discriminator or generator from drifting too rapidly.

Experimental Results: The Return of the King

GAT demonstrates a clear power-law relationship between compute and quality. In Head-to-head comparisons, the GAT-XL model outperforms established titans like StyleGAN-XL and GigaGAN.

• Data Efficiency: Reaches SOTA FID (2.18) in only 60 epochs.
• Inference Speed: Being a single-step (1-NFE) model, GAT-XL is ~200x faster than a 250-step DiT model.
• Versatility: Experiments show it works for ImageNet, FFHQ, and even text-to-image (MS-COCO) tasks.

A Look Behind the Layers

Ablation studies confirm the necessity of MNG. Without it, early layers show near-zero perceptual contribution (measured by LPIPS). With MNG, the responsibility for image synthesis is distributed more uniformly across the network.

Critical Insights & Future Work

The success of GAT highlights that the Discriminator's feature quality is the primary bottleneck for GAN scaling. By aligning the D features with Vision Foundation Models (e.g., DINOv2), the generator receives more informative gradients.

Limitations: While training is data-efficient, the per-iteration cost is higher than pure diffusion due to the D-updates. Future research into "Projected GAN" styles (using frozen pre-trained D-networks) could further slash training times.

In conclusion, GAT provides a robust roadmap for the next generation of single-step, high-fidelity generative models, proving that with the right scaling laws, adversarial learning is far from dead.