[CVPR 2024 candidate] ConfCtrl: Taming Video Diffusion with Kalman-Inspired Precise Camera Control

Executive Summary

TL;DR: ConfCtrl tackles the "hallucination vs. precision" trade-off in novel view synthesis. By treating noisy 3D point cloud projections as "measurements" in a Kalman-inspired feedback loop, it enables video diffusion models to follow extreme camera trajectories while maintaining flawless geometric consistency.

Background: In the landscape of 3D generation, we have two poles: Regression-based methods (like PixelSplat) which are geometrically accurate but fail at "filling the blanks," and Diffusion-based methods (like CameraCtrl) which are great at imagination but terrible at following the exact path you tell them to. ConfCtrl sits right in the middle, leveraging the generative power of diffusion with the rigorous control of 3D priors.

The Core Insight: Uncertainty is the Key

Most existing methods treat projected point clouds as ground truth conditioning. But if your depth estimation is off by 5%, your whole video warps. The authors of ConfCtrl realized that the model needs to know which parts of the 3D prior to trust.

By using confidence maps from 3D foundation models (like VGGT), they allow the diffusion process to lean on the point cloud where it's certain and drift into "generative mode" where the geometry is occluded or noisy.

Methodology: The "Kalman DiT" Revolution

1. Beyond Pure Noise Initialization

Instead of starting the denoising process from a blank slate of Gaussian noise, ConfCtrl starts from a Confidence-Aware Initialization. $z_0={\lambda }_1\cdot (\mathbf{w}\odot {\hat{z}}_{pc}^0)+{\lambda }_2\cdot ϵ$ This "warm start" ensures the model is already spatially aligned with the target geometry from step one, drastically reducing the search space for the diffusion flow.

2. The Predict-Update Architecture

The heart of the paper is the Kalman DiT Block. Inspired by the Kalman Filter’s recursive logic, it splits the control task into two stages:

• Prediction: The model takes the current latent and "projects" it forward based only on the target camera pose.
• Update: It then looks at the noisy point cloud projection and calculates a Residual Correction. If the prediction deviates from the 3D prior in a high-confidence area, the update step pulls it back.

Experiments & SOTA Results

The results are visually striking. Under large viewpoint changes where other models produce "melting" artifacts or lose the subject entirely, ConfCtrl maintains sharp boundaries and correct perspectives.

Quantitative Superiority

In benchmarks (CO3D, DL3DV), ConfCtrl consistently delivers:

• Lower Trajectory Drift: Translation and Rotation errors are reduced by ~20-30% compared to previous SOTA diffusion controllers.
• Better Detail Preservation: Thanks to a Latent Gradient Regularization loss ($L_{grad}$), fine structures don't get "washed away" during rapid camera movements.

Critical Analysis & Future Outlook

Why it works: Using a Video Interpolation backbone (Wan2.1) is a brilliant move. Unlike standard video generation, interpolation is naturally constrained by a "start" and "end" frame, providing a strong inductive bias for temporal stability that "camera-to-video" models lack.

Limitations: The reliance on a VAE (Variational Autoencoder) is still a bottleneck. Current VAEs are optimized for natural video textures, not for the surgical precision required in 3D reconstruction. As the authors suggest, "VAE-less" diffusion might be the next frontier for 3D-consistent video.

The Takeaway: ConfCtrl proves that "control" in generative AI isn't about forcing the model to obey; it's about teaching the model how to handle the uncertainty of the instructions it's given.