[CVPR 2025] S2D: Sparse to Dense Lifting—Redefining 3DGS Reconstruction with Minimal Inputs
S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs
S2D (Sparse to Dense) is a 3D reconstruction framework that enables high-fidelity 3D Gaussian Splatting (3DGS) from minimal inputs (even a single image). It leverages a novel one-step diffusion-based artifact fixer and a robust optimization strategy to achieve SOTA performance in view extrapolation and interpolation.
Executive Summary
TL;DR: S2D (Sparse to Dense) is a flexible framework designed to solve the "sparse-view" problem in 3D reconstruction. By combining the structural reliability of Point Clouds with the photorealistic generative power of Diffusion Models, it allows for high-quality 3D Gaussian Splatting (3DGS) using as few as 1 to 6 images.
Background: While 3DGS has revolutionized real-time rendering, it is notoriously "data-hungry." When inputs are sparse, the model overfits, creating blurry textures and broken geometry. S2D positions itself as a "bridge" between structural point clouds and high-fidelity Gaussians, setting a new SOTA for both in-the-wild and autonomous driving scenarios.
Problem & Motivation: The "Sparse View" Curse
Existing 3DGS methods fail under sparse inputs for two main reasons:
- Structural Collapse: Without enough viewpoints, the optimization cannot triangulate the correct 3D position of Gaussians, leading to "floaters."
- Guidance Inconsistency: Generative models (like GANs or standard Diffusions) can create pretty images, but they often "hallucinate" different details for the same object when viewed from different angles, destroying 3D consistency.
The authors' insight is simple yet powerful: Point clouds provide the "bones" (structure), while Diffusion provides the "skin" (texture). By lifting sparse points to dense Gaussian scenes, we can have the best of both worlds.
Methodology: The Artifact Fixer and Robust Optimization
The S2D pipeline consists of two primary stages:
1. Dual-Guidance Artifact Fixer
Instead of relying on a slow multi-step diffusion process, S2D uses a one-step diffusion model (fine-tuned from SD-Turbo). It uses a unique "Mixing Module" that takes:
- The Artifact Image: The current (broken) 3DGS render.
- Reference View: A nearby ground-truth image for texture.
- Point Cloud Render: Structural guidance from a VFM (like π3) to ensure the geometry is physically plausible.
2. Robust Reconstruction Strategy
To prevent the model from overfitting to the hallucinations of the diffusion fixer, S2D introduces:
- Random Sample Drop: Dynamically balances the weight between "real" input views and "fixed" novel views during optimization.
- Weighted Gradients: Uses a confidence mask derived from the point cloud. If a pixel has no corresponding point cloud support, its gradient is dampened, preventing the Gaussians from "learning" incorrect noisy details.
Experiments & Results: Crushing the Baselines
S2D was tested across four major datasets (3DOVS, RE10K, MIP360, and DL3DV).
Key Performance Metrics:
- Extreme Sparsity (1 View): On the 3DOVS dataset, S2D achieved a PSNR of 21.41, compared to 3DGS's 10.12 and DIFIX's 14.10.
- Driving Scenes: In Waymo Open Dataset lane-shift tests, S2D outperformed "StreetCrafter" in perceptual quality (FID), showing it can handle complex, moving urban environments.
The visual results (shown above) demonstrate that while other methods produce "ghostly" artifacts or inconsistent textures, S2D maintains sharp edges and consistent geometry even at 180°–360° view ranges.
Critical Analysis & Conclusion
Takeaway
S2D is a significant step toward "Universal 3D Reconstruction." Its ability to function as a "plugin" for various 3DGS backbones makes it highly practical for real-world applications like VR/AR and autonomous driving simulations.
Limitations
The method is still somewhat dependent on the quality of the initial Vision Foundation Model (VFM). As shown in the paper's "failure cases," if the VFM fails to generate a coherent point cloud (e.g., in extremely low-texture environments), the structural guidance becomes fragmentary.
Future Work
The authors suggest that the next frontier is a more unified architecture where spatial feature extraction and image-level fixing are integrated even more tightly, potentially eliminating the need for an external VFM altogether.
Author Perspective: S2D enables minimal input requirements for 3DGS applications, potentially allowing us to reconstruct entire rooms or street blocks from just a handful of snapshots.