One Pass Is Not Enough: Elevating Generative Diversity with Recursive Latent Refinement

TL;DR

The Recursive Token Mapper (RTM) replaces the traditional single-pass MLP mappers in generators with a recursive, iterative refinement process. By reusing a small shared block to progressively polish latent "style" codes, RTM achieves unprecedented levels of mode coverage and sample precision. It essentially solves the "diversity vs. quality" trade-off for one-step generators, outperforming several multi-step diffusion baselines in distribution recall.

The "FID Trap" and the Motivation for RTM

In the race for lower Fréchet Inception Distance (FID) scores, the research community has inadvertently incentivized "mode collapse." FID conflates fidelity (sharpness) and diversity (coverage). A model producing a handful of perfect, near-duplicate images can outscore a model that captures the full, messy tail of the real-world distribution.

The authors argue that Precision and Recall are the more critical metrics for the next generation of AI. Furthermore, they identify a structural flaw in current style-based architectures: the Mapper. In StyleGAN-like models, an MLP maps noise $z$ to style $w$ in one shot. This forces the network to decide everything—from global composition down to microscopic texture—in a single forward pass.

Methodology: The Power of Recursion

Rather than making the mapper wider or deeper (which risks overfitting and parameter explosion), RTM uses recursion.

1. Recursive Token Cycles

RTM adopts a nested structure of $L$ inner cycles and $H$ refinement steps.

• Inner State ($Z_L$): Updates rapidly to capture local features.
• Outer State ($Z_H$): Updates slowly to maintain global structure.
• Noise Re-injection: At every step, the original noise $Z_0$ is re-introduced, ensuring the refinement stays anchored to the intended latent identity.

2. Parameter Efficiency

Because the same block $f$ is reused across all $HimesL$ cycles, the model gains "effective depth" without adding parameters. An RTM with 32 iterations has roughly the same parameter count as a 2-layer MLP but the representational power of a 32-layer stack.

Figure 1: Comparison between the standard StyleGAN MLP mapper (left) and the Recursive Token Mapper (right).

Experiments and Results: Best of Both Worlds

The researchers tested RTM across two distinct training regimes: IMLE (non-adversarial, coverage-focused) and StyleGAN (adversarial, fidelity-focused).

• The IMLE Advantage: Combined with Rejection-Sampling IMLE (RS-IMLE), RTM hit the "triple intersection" of high quality, high diversity, and fast inference.
• SOTA Recall: On CIFAR-10, it achieved a Recall of 0.773, significantly higher than Diffusion models like EDM (0.618) and Flow-matching baselines.
• StyleGAN Integration: Simply swapping the mapper in StyleGAN2 improved FID and coverage, proving RTM is a "drop-in" upgrade for existing architectures.

Table 1: Performance on CIFAR-10. Note the superior Precision and Recall compared to diverse families of generative models.

Visual Evidence: Diversity in Action

The qualitative results demonstrate RTM's ability to cover the "long tail." In comparisons where a standard baseline might ignore specific genders, skin tones, or lighting conditions, RTM-based models consistently produce a more faithful and varied representation of the dataset.

Figure 2: RTM preserves subtle attributes (age, hair, skin tone) that are dropped by single-pass baselines.

Conclusion and Future Outlook

RTM proves that one pass is indeed not enough. By re-thinking the mapping network as an iterative refinement engine rather than a static transformer, we can build models that are not only sharp but truly representative of reality.

Takeaways for the Industry:

1. Metric Shift: Don't trust FID alone; prioritize Recall to ensure your product doesn't suffer from hidden mode collapse.
2. Recursive Bias: Parameter-sharing via recursion is a potent way to increase model "thinking time" without bloating the model size.
3. One-Step Future: High-quality, high-diversity generation doesn't require 50 diffusion steps—smart refinement in the latent space can achieve similar results in one pass.