[CVPR 2025] ArtiFixer: Bridging 3D Reconstruction and Auto-Regressive Video Diffusion
ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models
ArtiFixer is a novel framework that bridges 3D neural reconstruction and auto-regressive video generation to enhance sparse 3D Gaussian Splatting (3DGS). By distilling a bidirectional video diffusion model into a causal auto-regressive generator, it achieves state-of-the-art results in novel view synthesis, outperforming previous methods by 1-3 dB PSNR.
Executive Summary
In the realm of 3D vision, we have long faced a dichotomy: Neural Reconstruction (like 3DGS) offers high fidelity but fails in unobserved areas, while Video Generation (like Sora or Wan 2.1) creates plausible content but lacks precise geometric control.
ArtiFixer merges these worlds. It uses a corrupted 3DGS rendering as a scaffold for an auto-regressive video diffusion model. The result? A system that not only fixes artifacts in sparse 3D reconstructions but also "hallucinates" missing parts of the room with perfect temporal consistency. It outperforms existing SOTA by a massive 1-3 dB PSNR margin.
Problem & Motivation: The "Hole" in 3D Reconstructions
When you capture a scene with only a few photos, 3D Gaussian Splatting creates "floaters" or empty voids in areas the camera didn't see.
- The Scalability Trade-off: Previous methods used image-based diffusion, which requires slow, iterative "distillation-and-reconstruction" cycles to keep views consistent.
- The Quality Trade-off: Methods that condition too strongly on the "black holes" (unobserved pixels) often suffer from mode collapse, where the model just outputs black or blurred pixels instead of creatively filling the space.
Methodology: The Secret Sauce
ArtiFixer’s architecture is built on a two-stage evolution of the Wan 2.1 (14B) video model.
1. Opacity-Aware Noise Mixing
To solve the mode collapse problem, authors introduce Opacity Mixing. Instead of just feeding a degraded image to the model, they use the 3DGS opacity map (): In high-confidence areas (high opacity), the model stays true to the reconstruction. In "holes" (low opacity), it injects pure Gaussian noise, giving the diffusion model the "creative freedom" to inpaint a plausible floor, wall, or furniture.
2. Causal Distillation
Bidirectional models (that look at all frames at once) are too heavy for interactive use. ArtiFixer distills this into a Causal Auto-Regressive model. By applying a block-causal mask and a rolling KV cache, it can generate hundreds of frames in a single pass, using the noisy 3D rendering as a "guide rail" to prevent the typical drift seen in long-form generative video.
Experimental Results: Setting a New Standard
The performance leap is visible across all major benchmarks. On Mip-NeRF 360, ArtiFixer variants (including 3D-distilled versions) consistently exceed prior works like GenFusion and Cat3D.
Visual Fidelity
As shown in the DL3DV results, while other methods produce "ghostly" artifacts or lose color consistency, ArtiFixer maintains a sharp, photorealistic output even during long camera orbits.
Takeaway & Future Work
ArtiFixer proves that explicit 3D geometry is the best conditioning signal for generative models. By providing a "noisy map," we simplify the generative task from "creation from nothing" to "refinement of something."
Current Limitations: Although fast, it’s not yet "60 FPS" real-time. Future iterations will likely focus on single-step denoising and further reducing latency for VR/AR applications.
Senior Editor's Note: This paper effectively solves the "Sparse View" problem not by trying to make 3DGS better, but by acknowledging its flaws and using a world-class generative model to "fix" them auto-regressively.