[CVPR 2025] OrbitNVS: Redefining Novel View Synthesis as Orbital Video Generation

TL;DR

OrbitNVS transforms the challenge of Novel View Synthesis (NVS) from a geometric reconstruction problem into one of visual commonsense reasoning. By fine-tuning a powerful video generation model (Wan2.1) and equipping it with camera adapters, a normal map generation branch, and pixel-space supervision, the authors achieve a massive +2.9 dB PSNR boost over existing SOTA methods on single-image benchmarks.

The Core Insight: NVS as Visual Reasoning

Traditional NVS often tries to build an explicit 3D representation (like NeRF or Gaussian Splats) from sparse data. However, for a single input image, this is mathematically ill-posed. Humans solve this by using "visual commonsense"—we know what the back of a chair looks like because we've seen thousands of them.

OrbitNVS argues that video models are the best repositories for this commonsense knowledge. Unlike 2D image models, video models naturally understand temporal and spatial continuity.

Methodology: Bridging the Reality Gap

While video models have the "brains" for NVS, they lack the "tools." OrbitNVS introduces three critical components to turn a general video generator into a precise 3D tool:

1. Camera Adapters (The Steering)

To move beyond random video generation, the authors use Plücker coordinates to represent camera rays. These are fed into the Diffusion Transformer (DiT) via AdaLN (Adaptive Layer Normalization), allowing the user to dictate the exact orbital path of the camera.

2. Normal Map Guidance (The Skeleton)

RGB values entangle texture and geometry. To decouple them, OrbitNVS adds a parallel Normal Map branch. This branch shares parameters with the RGB branch and interacts via self-attention. By learning to predict the underlying shape (normals) alongside the surface (RGB), the model maintains much higher geometric fidelity.

Fig 1: The OrbitNVS Architecture showcasing the dual-branch RGB and Normal Map generation.

3. Pixel-Space Post-Training (The Clarity)

Latent Diffusion Models (LDMs) often lose fine details because the VAE compression is lossy. OrbitNVS introduces a second training stage where they back-propagate gradients through the VAE decoder. This "pixel-space" loss ensures the model learns to generate latents that reconstruct sharp, high-frequency details like text and fine patterns.

Experimental Performance

The results are striking. Across two major 3D datasets, GSO and OmniObject3D, OrbitNVS sets new records.

Qualitative Impact

The ablation studies highlight why these components matter:

• Normal branch: Without it, complex structures like woven baskets collapse into flat surfaces.
• Pixel-loss: Without it, barcodes and fine textures become an illegible blur.

Fig 2: Comparison showing the impact of Pixel-Space Loss on barcode legibility.

Critical Analysis & Future Outlook

Takeaway: OrbitNVS proves that "Common Sense > Geometry" when dealing with extreme data scarcity (single images). By leveraging a 14B parameter video foundation model, it bypasses the need for complex 3D caches.

Limitations:

1. Computational Cost: The two-stage training (latent then pixel-space) is resource-intensive.
2. Static Bias: The model is currenty specialized for static objects. Applying this to dynamic scenes with camera movement remains a challenge.

Future Work: Integrating even more powerful Vision-Language Models (VLMs) could allow for more semantic control—imagine asking the model to generate the "back of this chair but in a different color" while maintaining 3D consistency.