Chimera: Modernizing Visual Diffusion with Hybrid Attention and Principled Scaling

TL;DR

Visual generation is entering a "token-extensive" era where high-resolution images and long videos break the quadratic scaling of traditional Transformers. Adobe Research introduces Chimera, a hybrid backbone that blends Kimi Delta Attention (KDA) for linear-time processing with Multi-head Latent Attention (MLA) for global reasoning. By ditching positional embeddings in favor of modality-aware convolutions and utilizing a new scaling recipe called HeteroP, Chimera achieves a 7.3x compute efficiency gain and can generate 30-second videos despite being trained only on 5-second clips.

The Problem: The Quadratic Wall and Scaling Chaos

Modern visual models like Sora or Stable Diffusion 3 face two massive hurdles:

1. Architectural Cost: Full self-attention scales at $O(N^2)$. A single 4K image contains ~65K tokens; a video contains millions. This makes training and inference for long sequences nearly impossible on current hardware.
2. Scaling Uncertainty: Unlike LLMs, visual diffusion models haven't had clear "Chinchilla Laws." Research teams often guestimate how to scale model width vs. depth vs. data, leading to "undertrained" models that waste millions in compute.

Methodology: The Architecture of a Hybrid

Chimera treats text, images, and videos as a unified token sequence in a single-stream backbone.

1. Hybrid Attention (KDA + MLA)

Chimera uses a 3:1 ratio of KDA to MLA layers:

• KDA (Linear): Efficiently tracks state over long sequences with $O(N)$ complexity.
• MLA (Global): Periodically restores the ability to look at any token across the entire sequence using compressed KV caches to save memory.

2. NoPE (No Positional Embeddings)

Interestingly, the authors found that Rotary Positional Embeddings (RoPE) actually consume attention capacity and hinder extrapolation. Chimera replaces RoPE with:

• Modality-aware Short Convolutions: Text gets 1D causal kernels; Video gets 3D spatiotemporal kernels. These provide the necessary "local" bias.
• Causal Scan: The KDA recurrence itself provides order awareness.

HeteroP: A New Scaling Recipe

Scaling a heterogeneous model (containing linear attention, MoE, and convolutions) is harder than scaling a standard Transformer. A single global scaling ratio doesn't work because different modules have different "functional fan-ins."

HeteroP (Heterogeneous Parameterization) derives module-specific scaling ratios. This ensures that a learning rate tuned on a tiny 22M proxy model remains optimal for an 11B target model. This stability allowed the team to fit accurate Scaling Laws to predict exactly how much data and how many parameters are needed for a given compute budget.

Results: Efficiency and Extrapolation

The benefits of the Chimera approach are quantified in three major ways:

• Compute Efficiency: To reach the same diffusion loss, the full Chimera system is 7.3x more efficient than a full-attention baseline (Wan-2.1).
• Zero-Shot Extrapolation: Because it doesn't use positional embeddings tied to training length, Chimera can generate 30s videos (6x the training length) with almost no degradation in visual quality, whereas traditional models often "break" or stop moving.
• Benchmark Performance: Chimera (2.2B activated parameters) matches or beats much larger models like FLUX.1-dev on composition and text-alignment benchmarks (GenEval/DPG-Bench).

Critical Insights: Why it Works

The "Chimera" name is apt. The model succeeds by acknowledging that different tokens need different types of attention.

• Local interactions (texture, edges) are handled by convolutions.
• Long-range state (object persistence, motion flow) is handled by KDA.
• Global relationships (prompt alignment, layout) are handled by MLA.

By separating these roles into specialized modules rather than forcing a single Attention mechanism to do all three, Chimera frees up the model to learn more efficiently.

Conclusion and Future Outlook

Chimera proves that we don't need $O(N^2)$ attention for state-of-the-art visual generation. The introduction of HeteroP is a significant contribution to the field, offering a blueprint for how to scale diverse, complex architectures reliably. Future work will likely see these hybrid backbones becoming the standard for "World Models" that need to process minutes or hours of video context in a single stream.