RoboWM-Bench: Testing if AI World Models Actually Understand Physics

TL;DR

Can a robot actually "do" what an AI video model imagines? RoboWM-Bench is a systematic benchmark that proves the answer is often "No." By converting generated videos into executable robot actions within a high-fidelity simulator, researchers found that even the most "realistic" models (like Wan2.6 or Veo) frequently fail at basic physical tasks due to unstable contact points and spatial "hallucinations."

The Problem: The "Visual Realism" Trap

We've all seen the stunning videos generated by Sora or Kling—they look incredibly real. However, for a robot, "looking real" isn't enough. In the field of Physical AI, a video world model must yield a trajectory that obeys the laws of physics.

Prior work often suffered from two issues:

1. Perceptual Bias: Metrics like FID or PAI-Bench reward models for pretty pixels, not for whether a hand is actually touching a handle.
2. Scalability Issues: Validating videos on real robots is slow, expensive, and impossible to reproduce exactly across different labs.

Methodology: Bridging the Gap with Real-to-Sim

The core innovation of RoboWM-Bench is its unified execution pipeline. It doesn't just watch the video; it "replays" it in a physics engine.

1. Unified Action Extraction

The benchmark supports two pathways to turn pixels into movements:

• Human-Centric Retargeting: Uses pose tracking (HaMeR) and stabilized retargeting to map human hand movements in a video to a robot's end-effector.
• Robot-Centric Execution: Utilizes a Two-Stage Inverse Dynamics Model (IDM). By training on a mix of simulation data (large-scale) and real-world data (fine-tuning), the IDM can predict the exact joint actions required to match the video frames.

2. The Real-to-Sim Pipeline

To ensure evaluation is fair and reproducible, the authors use a 4D Gaussian representation to reconstruct the visual environment and 3D segmentation for interactive objects. This allows a video generated based on a real-world photo to be tested in a simulation that matches that photo's geometry.

Fig 1: The RoboWM-Bench Pipeline: from initial observation to simulation-based validation.

Experiments: A Reality Check for SOTA Models

The researchers tested heavyweights like Wan2.6, Veo3.1, and Cosmos. The results were a wake-up call for the community.

• Perceptual Scores are Deceptive: While models achieved nearly saturated scores on perceptual benchmarks (PAI-Bench), their execution success in RoboWM-Bench was significantly lower.
• The "Robot Structural" Problem: Interestingly, models are better at generating human hands than robot arms. Why? Because the internet has much more video of humans than of Franka robot arms. In generated robot videos, the arm often "warps" or deforms, leading to immediate execution failure.
• Complexity Kills: As tasks move from simple "Push Button" to "Assemble Burger" (long-horizon) or "Fold Towel" (deformable), success rates plummet to near zero for most models.

Table 1: Task-level success rates. Note the sharp drop in "Robot" tasks compared to "Human" tasks.

Deep Insight: Why do they fail?

The qualitative analysis highlights a phenomenon called "Visual Plausibility vs. Physical Infeasibility." In one example ("Put on Plate"), the generated video shows fingers merely touching a banana, and the banana mysteriously floats up. Visually, it looks okay. But when translated to a robot, there is no stable grasp, and the banana is immediately dropped in the simulator.

Fig 2: Visualizing the gap: The AI's imagined success (left) vs. the simulator's physical failure (right).

Conclusion & Future Work

RoboWM-Bench establishes that we cannot rely on general-purpose video models as "plug-and-play" simulators for robotics yet.

Moving Forward:

1. Embodied Fine-Tuning: As shown by "Cosmos-FT," even small amounts of task-specific data improve physics drastically.
2. Contact-Aware Generation: Future models need loss functions that penalize non-physical contact or structural deformation.
3. Real-to-Sim as Standard: Benchmarks like RoboWM-Bench should become the new standard for evaluating any "World Model" that claims to understand the physical world.

This work marks a shift from Video Generation to Video Foundation Models for Control, paving the way for truly autonomous agents that can plan their actions in their "mind's eye" before acting.