StereoWorld: Grounding Metric Geometry in End-to-End Stereo Video Generation

TL;DR

Researchers from HKU, VAST, and ByteDance have introduced StereoWorld, the first camera-conditioned stereo world model. Unlike representative monocular approaches that "hallucinate" depth, StereoWorld jointly learns appearance and binocular geometry. By utilizing a unified camera-frame RoPE and a geometry-aware attention mechanism, it generates view-consistent stereo videos 3x faster than traditional pipelines, enabling immediate applications in VR and embodied AI.

The "Geometry Gap" in World Models

Modern world models (predicting future observations based on actions) have reached high visual fidelity but typically operate in monocular RGB space. This creates a fundamental bottleneck:

• Scale Ambiguity: Monocular depth is relative, making metric-scale navigation for robots unreliable.
• Inconsistency: Generating a second view via post-hoc warping/inpainting often breaks fine details (e.g., thin wires or complex textures).
• Compute Overhead: Standard 4D attention (Space x Time x View) scales quadratically, making high-resolution stereo video synthesis nearly impossible for real-time use.

StereoWorld flips the script by treating Stereoscopy as the primary perceptual mechanism, grounding 3D structure directly in disparity.

Methodology: The Core Innovation

1. Unified Camera-Frame RoPE

Instead of concatenating ray encodings (which use absolute coordinates and fail to generalize), StereoWorld expands the feature dimension $d$ of a pre-trained DiT backbone. It adds an orthogonal, camera-conditioned subspace $d_c$. The rotary matrix is extended such that the original $dimesd$ block preserves pre-trained video priors, while the new $d_c$ block encodes the camera's intrinsic ($K$) and extrinsic ($T$) relationships. This "Copy-Init" strategy allows the model to reason about relative camera motion without destroying its ability to generate natural-looking video.

2. Stereo-Aware Attention Decomposition

The authors observe that in rectified stereo, correspondences align horizontally. To avoid the $O((2FHW{)}^2)$ cost of global attention, they decompose it:

• 3D Intra-view Attention: Handles temporal and spatial dynamics within a single eye.
• Horizontal Row Attention: Exchanges information across views only along the epipolar scanlines.

Figure 1: The StereoWorld architecture showing the injection of camera-aware RoPE and the stereo-aware attention block.

Experimental Performance

StereoWorld was benchmarked against SOTA models like Voyager and Aether combined with post-hoc stereo converters. The results are clear:

• Efficiency: 0.49 FPS vs Voyager's 0.03 FPS (a massive 16x jump).
• Consistency: Significant improvements in "View Synchronization" metrics (Mat. Pix. and CLIP-V).
• Geometry: Unlike RGB-D models that often bleed texture into depth maps, StereoWorld produces clean, disparity-aligned pairs that allow for accurate metric-scale reconstruction.

Table 1: Quantitative comparison showing superior visual quality, camera accuracy, and synchronization.

Beyond Benchmarks: VR and Embodied AI

VR Rendering

Because StereoWorld is end-to-end, it generates binocular clips ready for head-mounted displays (HMDs) without the artifacts common to depth-warping pipelines. Artifacts like the "hallucinated" sky or misaligned edges are effectively mitigated.

Embodied Scenarios

When fine-tuned on the DROID robotic dataset, the model acts as a metric-accurate world model for robot agents. It can predict the stereo outcome of a command like "put the lid on the teapot," allowing agents to "mentalize" the geometric outcome of their actions.

Long Video Distillation

By distilling the model into a causal-attention, 4-step sampling framework, the authors improved generation speed from 0.49 FPS to 5.6 FPS, showing a path toward real-time interactive environment simulation.

Figure 2: Disparity maps generated from StereoWorld outputs show much cleaner geometric boundaries compared to legacy RGB-D models.

Critical Insight & Conclusion

StereoWorld proves that explicit binocular awareness is more powerful than implicit depth estimation for 3D-consistent world modeling. By minimally perturbing pre-trained video priors through token expansion rather than re-parameterization, the model inherits the vast knowledge of internet-scale video datasets while gaining the surgical precision of stereo geometry.

Limitations: The model still reflects the scarcity of dynamic stereo training data, leading to a bias toward static environments. However, as robot fleets and stereo-rigged vehicles capture more data, the StereoWorld architecture is perfectly positioned to scale.