World Reconstruction From Inconsistent Views: Turning Video Diffusion into 3D Reality

TL;DR

Video Diffusion Models (VDMs) are great at imagining worlds but terrible at maintaining 3D "truth"—objects warp and drift as the camera moves. This paper introduces a sophisticated reconstruction pipeline that uses non-rigid alignment and inverse deformation rendering to neutralize these errors. The result? High-fidelity, explorable 3D worlds generated from standard, inconsistent AI videos.

The Problem: The "Generative Drift" Trap

If you've ever watched an AI-generated video closely, you'll notice that a table might subtly change shape or a wall might shift as the "camera" pans. In the vision community, we call this generative drift.

When we try to turn these videos into 3D models using standard tools like 3D Gaussian Splatting (3DGS), these tiny inconsistencies become catastrophic. The sea of Gaussians tries to "average out" the conflicting positions of the same object, leading to:

• Blurry textures (the model can't decide where the surface is).
• Floating artifacts (extra geometry created to explain the drift).
• Low consistency (the scene falls apart when viewed from a new angle).

Methodology: The Non-Rigid Fix

The authors argue that we shouldn't wait for "perfect" video models. Instead, we can treat the video's inconsistencies as a non-rigid tracking problem.

1. Lifting to Point Clouds

Using Geometric Foundation Models (like DepthAnything-3), the system lifts every video frame into a 3D point cloud. Predictably, these point clouds don't match up; the surfaces are "thick" and multi-layered due to inconsistencies.

2. Tailored Non-Rigid ICP

To squash these layers into single, sharp surfaces, the authors use a Non-rigid iterative frame-to-model ICP.

• Neural Deformation Fields: A hashgrid MLP predicts "twists" (local rigid transformations) for every point.
• Sparse Correspondences: They use the RoMa matcher to find long-range links between frames, preventing the alignment from getting stuck in local optima.

Figure 1: The pipeline from inconsistent VDM frames to a unified, sharp 3D point cloud and final Gaussian Splatting reconstruction.

3. Non-Rigid Aware Gaussian Splatting

This is the "secret sauce." Instead of forcing the 3D model to fit the messy images, they perform Inverse Deformation Rendering. They optimize the 3D Gaussians in a clean "canonical" space but temp-warp them back into the "messy" space of the original frames just for the loss calculation. This factors out the drift, allowing the model to learn sharp details without the geometric noise.

Results: Sharpness Where It Counts

The improvements are visually striking. While previous SOTA methods like VGGT-X produce "ghostly" or sparse reconstructions, this method yields dense, surface-aligned models.

Figure 2: Comparison of point cloud quality. Notice how the proposed method (bottom) unifies overlapping surfaces into a single, clean geometry compared to DA3 or VGGT-X.

In quantitative benchmarks (WorldScore), the approach achieves a Photometric Consistency of 86.59, dwarfing the 65.58 achieved by rigid-only methods. It effectively bridges the gap between 2D video quality and 3D structural integrity.

Critical Insight: The Future of World Models

The most profound takeaway here is the shift in responsibility. Traditionally, we've tried to force the Generator (the VDM) to be more consistent through costly fine-tuning on 3D data. This paper suggests that the Reconstructor can be smart enough to handle the generator's "lies."

Limitations

• Hallucinations vs. Drift: If the VDM radically changes an object (e.g., a chair turns into a plant upon revisiting), alignment can't save it. The method handles warping, not replacement.
• Compute Overhead: The alignment stage takes about 25 minutes per scene—a potential bottleneck for real-time applications.

Conclusion

By treating AI video drift as a non-rigid deformation problem, Höllein and Nießner have provided a robust bridge between the "dreamy" world of diffusion and the "rigid" world of 3D geometry. This is a vital step toward creating massive, explorable digital twins from nothing but a text prompt.