[CVPR 2025] GIFSplat: Bridging the Gap Between Feed-Forward Efficiency and Diffusion-Guided Refinement

TL;DR

3D reconstruction traditionally faces a trade-off: the high-fidelity but slow iteration of per-scene optimization vs. the lightning-fast but low-detail one-shot feed-forward prediction. GIFSplat shatters this dichotomy. By introducing an Iterative Feed-Forward mechanism and distilling Diffusion Priors into lightweight Gaussian cues, it achieves SOTA visual quality in seconds, even with sparse, unposed input views.

The Motivation: Why One-Shot is Not Enough

Recent feed-forward models (like PixelSplat or MVSplat) have made 3DGS nearly instantaneous. However, they suffer from two fatal flaws:

1. Capacity Bottleneck: A single forward pass often fails to capture the intricate details of complex scenes.
2. Lack of Scene Adaptation: Without an optimization loop, the model cannot correct its own residual errors based on rendering evidence.

While generative priors (diffusion models) could help, typical "Reconstruction-via-Generation" loops are computationally heavy, requiring thousands of gradient steps or a massive expansion of the reference view set.

Methodology: The "Iterative" Breakthrough

GIFSplat introduces a three-tier architecture:

1. Gaussian Initializer

The journey begins by removing voxelization constraints from existing baselines to predict camera poses and an initial set of 3D Gaussians ($G^{(0)}$).

2. Iterative Gaussian Head

Instead of stopping at $G^{(0)}$, GIFSplat applies $T$ forward-only update steps.

• Observation Cues ($o_i$): It renders the current state, computes the difference from the input image in feature space, and pools these errors back to the Gaussians.
• Residual Updates: A weight-shared head predicts $\Delta G$ to refine position, scale, color, and opacity.
• Crucially: This is a pure forward pass—no backpropgation is used during inference.

3. Generative Prior Fusion

To fill in the gaps where input views are too sparse, GIFSplat uses a frozen DIFIX diffusion enhancer. It enhances the current synthesis, extracts the "improvement delta" in feature space ($p_i$), and injects this as a guiding cue into the next iteration.

Experimental Results: SOTA Performance

GIFSplat was tested rigorously on DL3DV, RealEstate10K, and the out-of-domain DTU dataset.

• Fidelity: In 8-view scenarios (DL3DV), it achieved a PSNR of 24.91, significantly outperforming AnySplat (23.76).
• Sparse View Robustness: On 2-view RealEstate10K, the iterative refinement significantly reduced "texture sticking" and artifacts in under-constrained regions.
• Inference Speed: Despite the iterative nature, it maintains second-scale performance, offering a 100x speedup over optimization-based diffusion methods.

Ablation Insight: The Power of Iteration

As shown in the authors' studies, the transition from Step 0 to Step 3 provides a monotonic improvement in visual clarity. Sharp edges (e.g., door frames) and consistent textures emerge through these residual updates.

Critical Analysis & Future Outlook

GIFSplat establishes a new paradigm: Iterative Feed-Forward. It proves that we can mimic the "intelligence" of an optimization loop within a fixed-time neural network execution.

Limitation: Currently, the refinement focuses on static scenes. While it handles unposed views brilliantly, the extension to dynamic 4D scenes (video-to-3D) remains a challenge due to the temporal consistency required by the diffusion prior.

The Bigger Picture: This work paves the way for high-fidelity AR/VR content creation on mobile devices, where the second-scale inference and sparse-view robustness are critical for user experience.