[CVPR 2025] SimRecon: Bridging the Real-to-Sim Gap with Active Viewpoint Optimization and Scene Graph Synthesis

TL;DR

SimRecon transforms standard 2D videos of cluttered indoor environments into interactive, simulation-ready 3D scenes. By treating reconstruction as a "Perception-Generation-Simulation" pipeline, the authors solve the twin problems of visual infidelity and physical implausibility. The secret sauce lies in Active Viewpoint Optimization (AVO) for better object generation and a Scene Graph Synthesizer (SGS) that acts as a blueprint for stable physical assembly.

Background: The Limits of Holistic Reconstruction

Most recent breakthroughs in 3D vision, such as 3D Gaussian Splatting (3DGS), focus on "holistic" scenes. They create beautiful "digital dioramas" that look great from a camera but are functionally useless for robotics. If you want a robot to pick up a kettle, the model needs to know where the kettle ends and the table begins.

Existing compositional methods attempt to solve this but fail in complex, cluttered scenes. They often select "bad" views (heavily occluded) for object completion, leading to "melted" geometry, and they lack a sense of physical grammar, resulting in objects that float in mid-air or clip through walls when imported into simulators.

Methodology: The Core Bridging Modules

SimRecon introduces a structured set of object primitives, each defined by intrinsic (geometry, textures, mass) and relational (supportive, attached) attributes.

1. Active Viewpoint Optimization (AVO)

Instead of picking a random frame from the video, AVO "hunts" for the best viewpoint in 3D space.

• The Intuition: A good view is one that reduces information entropy.
• The Implementation: The authors use the accumulated opacity from the Gaussian Splatting pipeline as a differentiable proxy for information gain. By maximizing this opacity, the system automatically finds the angle that sees the "most" of the object with the least occlusion.

Figure 1: The SimRecon pipeline featuring AVO and SGS modules.

2. Scene Graph Synthesizer (SGS)

To avoid the "floating chair" problem, SimRecon builds a 3D Scene Graph.

• Progressive Merging: It clusters objects into regions, uses a VLM (like Qwen2.5-VL) to identify relationships (e.g., "cup is supported by table"), and merges these local graphs into a global hierarchy.
• Hierarchical Assembly: Unlike previous methods that use random search to fix collisions, SimRecon mimics the "real world" construction. It places the floor, then the table on the floor, then the cup on the table, using a brief physics simulation at each step to let gravity settle the objects naturally.

Experiments & Results

The authors tested SimRecon on the ScanNet dataset against SOTA baselines like InstaScene and DPRecon.

• Geometric Accuracy: SimRecon achieved a Chamfer Distance of 4.34, significantly lower than InstaScene's 6.90, proving that better view selection leads to more accurate 3D shapes.
• Visual Quality: The LPIPS score (measuring perceptual similarity) dropped to 0.153, nearly half of the competing methods.
• Physical Validity: In simulation tests (Blender/Isaac Sim), SimRecon's scenes remained stable, whereas MetaScenes—which relies on post-hoc collision fixing—often failed to model accurate contact relationships.

Figure 2: Qualitative results showing SimRecon's ability to reconstruct individual objects faithfully compared to baselines.

Critical Analysis & Conclusion

Takeaway

SimRecon’s biggest contribution is the realization that compositional reconstruction is a bridge between computer vision and physical simulation. By using differentiable optimization to select generation conditions (AVO) and semantic graphs to guide assembly (SGS), it provides a robust template for "Real-to-Sim" pipelines.

Limitations & Future Work

• Inference Time: While faster than SDF-based methods (DPRecon), the pipeline still takes ~21 minutes per scene. Real-time application is not yet feasible.
• Scalability: The current graph merging relies on VLMs, which might struggle with extremely dense scenes (hundreds of small objects) without better spatial grounding.
• Outlook: This work paves the way for automated creation of diverse training environments for Embodied AI agents directly from casual mobile video.

Author Perspective: By shifting from "holistic rendering" to "constructive assembly," we move 3D reconstruction closer to the native principles of the physical world.