Mirage: Revolutionizing Video World Models with Latent Spatial Memory

TL;DR

Maintaining 3D consistency in video generation usually requires heavy RGB-based point clouds that slow down systems and blur details. Mirage changes the game by caching scene information directly in the VAE latent space. This "Latent Spatial Memory" eliminates the need to render pixels and re-encode them at every step, resulting in 10x faster generation, 55x less memory usage, and far superior geometric stability in long-form videos.

The Bottleneck: The Paradox of Pixel-Space Memory

Most modern video world models (like Sora or Wan2.1) are phenomenal at generating short clips but struggle with "geometric drift" over long trajectories. To fix this, researchers previously added a 3D "brain" in the form of an RGB point cloud.

However, this created two massive bottlenecks:

1. Computational Drag: At every single frame, the system had to rasterize millions of points into a high-res image and then run a VAE encoder to turn those pixels back into latents.
2. Information Loss: The VAE encoder/decoder pair is lossy. Round-tripping features through pixel space discards the rich, high-dimensional nuances the model needs to stay consistent.

Methodology: Thinking Directly in 3D Latents

Mirage’s core insight is simple yet profound: If the model works in latent space, the memory should stay in latent space.

1. Latent Back-Projection

Instead of saving RGB values, Mirage takes the latent tokens ($16imes16$ compression) and "lifts" them into 3D coordinates using depth estimation. Each point in the world-space cache now carries a high-dimensional latent vector ($C=48$ channels) directly from the VAE.

2. Efficiency by Design

Because the memory is stored at the latent resolution (stride $s=16$), the number of points is reduced by a factor of $16^2=256$ compared to pixel-space points.

3. Occlusion-Aware Readout

When the "camera" moves to a new position, Mirage projects the 3D latent cache onto the new view. It uses z-buffering to handle visibility—ensuring that foreground objects correctly hide background elements—and passes the resulting feature map to a ControlNet-style branch of the diffusion transformer.

Figure: The Mirage cycle. Unlike prior works that detour through pixels, Mirage stays inside the VAE manifold for construction and readout.

Experiments: Speed Meets Stability

Mirage was tested against heavyweights like Spatia, Voyager, and foundation models like Wan2.1.

SOTA Performance

On the WorldScore benchmark, Mirage outperformed all baselines in 3D Consistency and Photometric Consistency. Most impressively, it solved the "loop closure" problem: when a camera travels in a circle and returns to its starting point, Mirage remembers the original scene perfectly, whereas other models hallucinate new, incompatible details.

The Efficiency Gap

The most striking result is the efficiency gain. By avoiding the pixel-space detour, Mirage scales linearly and gracefully.

• Speed: 10.57x faster than RGB-cache pipelines.
• Memory: 55x reduction in cache footprint.

Figure: Comparison of per-frame read time and memory usage. Latent spatial memory (orange) remains nearly flat while RGB methods (blue) explode.

Critical Analysis & Takeaways

Mirage proves that the "bottleneck" in world modeling wasn't the 3D representation itself, but the interface between 3D geometry and the neural latent space.

Key Takeaways:

• Geometric-Semantic Synergy: Latent tokens carry semantic clusters (walls, windows) better than raw RGB, providing a stronger "scaffold" for the generator.
• Dynamic Filtering is Crucial: By excluding dynamic objects (cars, people) from the persistent cache, Mirage avoids "ghosting" artifacts where moving objects get stuck in the 3D map.

Limitations: Currently, Mirage treats the world as mostly static. Reconstructing and caching the state of dynamic agents (e.g., a person walking into a room) remains a future challenge.

Conclusion

Mirage sets a new standard for efficient, consistent video generation. By treating 3D memory as a native part of the latent manifold, it enables us to simulate worlds that are not only visually stunning but geometrically rock-solid.