BetterScene: Breaking the VAE Bottleneck for Photorealistic 3D Scene Synthesis
BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model
BetterScene is a novel 3D scene synthesis framework that enhances Novel View Synthesis (NVS) from sparse photos by integrating feed-forward 3D Gaussian Splatting (3DGS) with a representation-aligned Stable Video Diffusion (SVD) model. It achieves state-of-the-art visual quality on the DL3DV-10K benchmark through a specialized VAE architecture that preserves high-frequency details.
TL;DR
The quest for photorealistic 3D scene synthesis from just a handful of photos has long been plagued by blurry "ghosting" artifacts. BetterScene tackles this by rethinking the latent space of video diffusion models. By introducing a representation-aligned and equivariance-regularized VAE with 64 latent channels, it allows a Stable Video Diffusion (SVD) backbone to "imagine" missing details with unprecedented consistency and high-frequency fidelity.
Background Positioning
In the landscape of 3D vision, we have moved from implicit representations (NeRF) to explicit primitives (3D Gaussian Splatting). However, both struggle with "sparse-view" settings. BetterScene sits at the intersection of Generalizable 3DGS and Generative Enhancement, functioning as a powerful "denoising polisher" that fixes the geometric and textured failures of feed-forward 3D models.
Problem: The VAE Optimization Dilemma
Why do current diffusion enhancers still produce "jittery" or "mushy" details? The authors point to a fundamental bottleneck in the Latent Diffusion Model (LDM) pipeline: the VAE.
- Lower Dimensionality: Standard VAEs (like the one in Stable Diffusion) use only 4 channels, which is too narrow to "bottle" the complex textures of real-world 3D scenes.
- The Dilemma: Increasing channel dimensions usually improves reconstruction but breaks the generative capabilities (making the model fail to create new content).
- Alias & Shift Sensitivity: Standard latents are not equivariant; a tiny shift in the camera path can cause a massive, inconsistent change in the generated texture.
Methodology: High-Dimensional Alignment
BetterScene introduces a two-stage training process to solve these issues.
1. The BetterScene-VAE
Instead of a narrow 4-channel latent space, the authors scale up to 64 channels. To prevent the generative performance from collapsing, they use:
- DINOv2 Alignment: They force the VAE latents to align with DINOv2 (a vision foundation model) features. This ensures the latent space is rich in semantic and structural information.
- Equivariance Loss: They penalize the model if a transformation (like a rotation or shift) on the input image doesn't produce a mathematically equivalent transformation in the latent space.
2. The Enhancement Pipeline
- Stage 1: An MVSplat model generates a "coarse" 3DGS representation and a rendered view.
- Stage 2: These coarse features are fed into the SVD U-Net as conditioning. Our optimized VAE then decodes the final, high-fidelity frames.
Experiments & Results: Real-World Dominance
The model was tested on the DL3DV-10K dataset—a massive collection of real-world, unbounded scenes.
- Visual Fidelity: BetterScene excels at recovering sharp textures (like text on a wall) that previous SOTA models like MVSplat360 either blurred or ignored.
- Consistency: Thanks to the equivariance regularization, the synthesized "video" of the 3D scene remains stable as the camera moves.
As shown in the table below, the jump in latent capacity (from C=4 to C=64) leads to a massive leap in PSNR and FID scores:
| Latent Channels (C) | PSNR ↑ | SSIM ↑ | rFID ↓ |
|---|---|---|---|
| Standard SD-VAE | 26.06 | 0.78 | 13.83 |
| BetterScene (C=64) | 31.21 | 0.92 | 4.90 |
Critical Analysis & Conclusion
Takeaway
BetterScene proves that the "secret sauce" for high-fidelity 3D generation isn't just a bigger U-Net, but a wider latent bottleneck. By aligning generative latents with foundation models, we can leverage huge amounts of prior visual knowledge to solve the ill-posed problem of sparse reconstruction.
Limitations
The primary hurdle remains computational cost. Training a customized SVD pipeline with high-dimensional latents requires significant GPU resources (the authors used four H100s). Future iterations could benefit from more efficient "Linear Diffusion Transformers" or distilled architectures to make this real-time on consumer hardware.
Future Outlook
The success of "Equivariance Regularization" here has broad implications. We are likely to see this technique applied to other temporal-sensitive tasks like video editing and robotic vision, where spatial stability is non-negotiable.