[CVPR 2024] FrameDiT: Solving the Motion-Alignment Paradox in Video Generation with Matrix Attention

TL;DR

FrameDiT introduces a breakthrough "Matrix Attention" mechanism that treats video frames as holistic matrices rather than isolated tokens. It successfully bridges the gap between the expensive Full 3D Attention (great quality, poor speed) and Local Factorized Attention (great speed, poor motion handling). By enabling a global "frame-to-frame" dialogue, FrameDiT-H achieves SOTA performance on major benchmarks while remaining as efficient as traditional factorized DiTs.

The Bottleneck: Why Your AI Videos "Drift"

In the current landscape of Video Diffusion Transformers (DiTs), researchers usually pick one of two poisons:

1. Full 3D Attention: It looks at every token in every frame simultaneously. It's beautiful and coherent, but the $O(T^2{N}^2)$ complexity means your GPU will run out of memory the moment you try to generate a high-res, long video.
2. Spatially Local Factorized Attention: This is the industry standard (used in models like Latte). It processes space, then processes time at the exact same pixel location.

The Insight: If an actor moves from the left of the screen to the right, Local Factorized Attention loses them. It expects the "temporal" magic to happen at the same $[x,y]$ coordinate. When motion is large, the temporal coherence collapses because the model has no global spatial awareness across frames.

Methodology: Entering the Matrix

To solve this, the authors propose Matrix Attention. Instead of attending to individual tokens, the model attends to frames as a whole.

Matrix-Native Operations

For each frame $z^t$, the model computes Query, Key, and Value matrices using learnable row-weight ($U$) and column-weight ($W$) matrices: $q^t=U_q^{o}p{z}^t{W}_q+B_q$ The row-weight matrix $U$ is the secret sauce—it acts as a "learnable compressor" that summarizes the $N$ spatial tokens into a condensed frame representation.

Figure: The FrameDiT architecture showing the interleaving of Spatial and the new Matrix Temporal Attention blocks.

The Hybrid Advantage (FrameDiT-H)

The authors realized that while Matrix Attention captures global motion, we still need local attention for fine-grained details (like textures). FrameDiT-H uses two parallel branches:

• Local Branch: Standard temporal attention (for pixel-level consistency).
• Global Branch: Matrix Attention (for object-level tracking across the frame).

Experiments: Efficiency Meets Quality

When scaling to long videos (up to 128 frames), FrameDiT proves its worth. While Full 3D Attention hits a "memory wall," FrameDiT maintains the low memory footprint of factorized models but keeps the high FVD (Fréchet Video Distance) quality of the heavyweights.

Figure: Comparison of FVD, FLOPs, and Memory. Note how FrameDiT variants (red/orange) track the efficiency of Local Factorized models while matching the quality of Full 3D.

Qualitative SOTA

On the FaceForensics dataset, FrameDiT-H smashed the previous records, achieving an FVD of 16.6 (nearly 40% better than Latte's 27.1). It effectively eliminates the "identity shifting" and "structural collapse" common in other efficient models.

Conclusion and Future Outlook

FrameDiT proves that we don't need to choose between speed and motion coherence. By rethinking temporal attention as a matrix-native operation, the authors have provided a scalable blueprint for the next generation of video models.

Limitations: Currently, the model relies on a frozen VAE (Stable Diffusion 2.0), which can limit the reconstruction of very fine details like fingers or eyes in low-resolution settings. Future work involving end-to-end training with video-native VAEs could push these boundaries even further.

Takeaway: If you are building video models, look beyond token-to-token attention. The "Frame-as-a-Matrix" perspective is the key to handling complex dynamics without breaking the computational bank.