[Project DVD] Deterministic Video Depth: Unlocking Generative Priors with 1% of the Data

TL;DR

Video depth estimation has long been stuck between two worlds: Generative models that "hallucinate" inconsistent geometry, and Discriminative models that require mountains of data to avoid "blurring" boundaries. DVD (Deterministic Video Depth) breaks this cycle. By turning a pre-trained video diffusion model (WanV2.1) into a single-pass regressor, the authors achieve SOTA zero-shot results using 163x less data than previous leaders like Video Depth Anything.

The "Ambiguity-Hallucination" Dilemma

Why is video depth still hard?

1. Stochastic Hallucination: Generative models use multi-step sampling. Every "step" is a chance for the model to invent a slightly different floor or wall, leading to flickering and "geometric drift" in long videos.
2. Semantic Ambiguity: Discriminative models (like ViTs) learn strictly from data. If they encounter motion blur or a textureless white wall, they lose the "intuition" of what the object should look like, resulting in messy boundaries.

DVD's core insight is simple yet profound: Keep the generative "smart" priors of a world-simulator model, but ditch the stochastic "sampling" process.

Methodology: The Three Pillars of DVD

1. Timestep as a Structural Anchor

In typical diffusion, the timestep $t$ tells the model how much noise to remove. In DVD, $t$ is no longer a clock—it's a tuning knob for frequency.

• High $t$ (0.8): The model focuses on global, low-frequency shapes (stable but blurry).
• Low $t$ (0.0): The model focuses on high-frequency edges (sharp but unstable). DVD anchors the model at an optimal $au=0.5$, balancing "The Big Picture" with "Fine Details."

2. Latent Manifold Rectification (LMR)

Standard regression often suffers from "mean collapse"—the model plays it safe and predicts a blurry average of possible depths. DVD fixes this by supervising the differentials (the change between pixels and frames) directly in the VAE latent space. This forces the model to maintain sharp spatial boundaries and smooth temporal flow.

3. Global Affine Coherence

For long videos, we must use sliding windows. Generative models usually fail here because their scales don't match between windows. DVD discovers that its deterministic backbone maintains Affine Coherence: the difference between windows is just a simple linear scale and shift. Use a closed-form least-squares solver, and voila—perfectly aligned long-video depth.

Experimental Results: Doing More with Less

The most "shocking" result is data efficiency. Video Depth Anything (VDA) was trained on 60 million frames. DVD beats it using only 367,000 frames.

DVD isn't just more accurate; it's more "knowledgeable" because it inherits the world-modeling capabilities of the foundation diffusion model it was built upon.

Critical Perspective & Conclusion

Limitations: While DVD is a massive leap forward, it still uses a heavy DiT backbone (1.3B parameters). Real-time 60FPS deployment on a smartphone isn't here yet. Also, as seen in the "Failure Case" analysis (Figure 14), extreme scene transitions (like moving from a dark room to a bright tunnel) can still cause absolute scale drift, though local geometry remains sharp.

The Takeaway: DVD proves that the future of 3D vision isn't just "more data." It's about "better adaptation." By treating diffusion models as deterministic world-priors, we can achieve SOTA results with a fraction of the training cost.

Find the code and models at: https://github.com/EnVision-Research/DVD