[CVPR 2025] RnG: Unifying 3D Reconstruction and Generation via Causal Transformers
RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations
RnG (Reconstruction and Generation) is a unified feed-forward Transformer designed for complete 3D modeling from sparse, unposed 2D images. It achieves state-of-the-art results by simultaneously performing implicit 3D reconstruction and high-fidelity novel view synthesis (RGBD), outperforming Matrix3D and VGGT in both geometric accuracy and inference speed.
TL;DR
RnG (Reconstruction and Generation) is a breakthrough Transformer-based framework that solves the "incomplete geometry" problem in 3D reconstruction. Unlike previous models that only reconstruct what they see, RnG can "see the unseen" by unifying reconstruction and novel-view generation. It runs in real-time (<0.1s) on an A800 GPU and generates complete 3D objects from just a few unposed images.
Motivation: The "Partial Obsession" of 3D Vision
Current 3D foundation models, while impressive at geometry recovery, suffer from a fundamental flaw: they are essentially "3D mirrors" of their input. If a camera doesn't see the back of a chair, the model doesn't build it. This lack of hallucination capability makes current outputs unsuitable for physics simulations or 3D assets that require 360-degree integrity.
While generative models (like Diffusion-based Matrix3D) can imagine these missing parts, they are computationally expensive and often lack strict 3D consistency. The authors of RnG asked: Can we use the latent 3D understanding already present in reconstruction models to drive high-speed generation?
Methodology: The Power of Causal Attention
The core innovation is the Reconstruction-Guided Causal Attention. In a standard Transformer, all tokens attend to each other. RnG introduces a binary mask that ensures:
- Source tokens (input images) only attend to other source tokens (Stabilizing the reconstruction).
- Target tokens (query viewpoints) attend to both source and target tokens (Leveraging the reconstructed features for generation).
KV-Cache as the "Virtual 3D Scanner"
By utilizing the causal design, the model's KV-cache becomes a persistent, implicit 3D representation.
- Step 1 (Reconstruct): Processes the source images once and stores the Key/Value pairs.
- Step 2 (Generate): To render a new view, the model simply queries the KV-cache with a new camera pose (Plücker rays). This ignores the need to re-process input images, slashing inference time from 213ms to just 85ms.
Experimental Breakthroughs
RnG was tested on benchmarks like Google Scanned Objects (GSO). It managed to surpass specialized models in their own domains:
- Geometry: Achieved a Chamfer Distance an order of magnitude better than VGGT by creating "closed" 3D models rather than just "visible shells."
- Speed: 100x faster than Matrix3D, making it viable for interactive 3D scanning on mobile or edge devices.
Visual comparison shows how RnG (right) provides photorealistic and structurally sound "backsides" of objects compared to the noisy or incomplete outputs of previous SOTA methods.
Critical Analysis & Conclusion
RnG represents a shift from "reconstruction vs. generation" to a unified latent perception. By treating novel view synthesis as a query into a pre-computed 3D memory (KV-cache), it bridges the gap between efficiency and completeness.
Limitations:
- Detail Level: It lacks the hyper-fine textures often found in Diffusion models.
- World Origin: The model assumes cameras look toward a common center, which may limit unstructured "in-the-wild" scans without further data augmentation.
In summary, RnG proves that Transformers don't just need to predict the next token—they can predict the next dimension, turning partial 2D views into complete 3D realities.