[CVPR 2025] Geometry-as-Context: Solving the Consistency Crisis in 3D Video Generation

TL;DR

Researchers from Peking University and TeleAI have introduced Geometry-as-Context (GaC), a framework that treats 3D geometry as a learnable context within a video generator. By replacing "hard" non-differentiable reconstruction steps with a "soft" differentiable DiT (Diffusion Transformer), they've created a system that generates scene-consistent videos with unprecedented camera accuracy.

The "Butterfly Effect" in 3D Reconstruction

Why do most AI-generated videos look like a fever dream when the camera turns around? Existing reconstruction-based methods usually follow a rigid loop:

1. Estimate depth.
2. Unproject to a 3D point cloud (Non-differentiable).
3. Render a new view.
4. Inpaint the holes.

The problem is that if Step 1 is off by even a tiny margin, the error compounds at Step 2 and 3. Since the rendering is non-differentiable, the model can't "learn" from its mistakes. This leads to the cumulative error or "butterfly effect," where the scene becomes increasingly garbled over time.

The GaC Insight: Everything is a Token

GaC solves this by turning the 3D pipeline into a single, end-to-end differentiable sequence. Instead of using external software to render, they train a DiT to handle interleaved sequences of RGB images and Geometry (Depth) maps.

1. Model Architecture: Camera Gated Attention (CGA)

Standard models often just "add" camera poses as a hint. GaC introduces CGA, which uses Plücker rays to modulate the Self-Attention mechanism. It doesn't just ask "what is the pose?"; it uses a gating matrix to decide how that pose should influence geometry prediction versus texture synthesis.

2. Geometry Dropout: Learning without Bloat

Training on double the frames (RGB + Depth) is slow. To fix this, the authors use Geometry Dropout. During training, they occasionally drop the geometry frames. This forces the model to learn 3D consistency when the geometry is present, but remain capable of high-speed RGB-only generation during inference.

Experimental Battleground

The model was put to the test on RealEstate10K and Tanks-and-Temples datasets.

• Visual Fidelity: GaC achieved an FID of 55.76, crushing the previous SOTA of ~80.
• Camera Precision: Rotation and translation errors dropped significantly, meaning the AI follows the user's requested path with surgical precision.
• The Cyclic Test: In "forth-and-back" trajectories (moving the camera away and returning to the start), GaC successfully "remembered" objects that had disappeared from view—a feat most video models fail at.

Critical Analysis & Conclusion

Why it works

By unifying 3D geometry and image synthesis, the model develops an internalized world model. It "understands" that a pixel isn't just a color, but a point in space. The use of a single DiT (based on Bagel-7B) allows for cross-modal reasoning that separate models simply cannot achieve.

Limitations

As noted in the supplement, the model still struggles with:

• Complex Dynamic Subjects: It excels at static scenes but can falter with moving humans.
• Boundary Artifacts: Some darkened textures appear at the edges of extreme FOV changes.

The Future of GaC

The transition from "Geometry as a Tool" to "Geometry as a Context" is a major shift. It paves the way for interactive 3D environments for VR/AR and embodied AI, where consistency isn't just a preference—it's a requirement for functional navigation.

Takeaway: If you want a consistent world, don't just generate images; generate the logic of the space they inhabit.