[CVPR 2026] Dream2Act: Bridging the Morphology Gap via Robot-Centric Video Hallucination

TL;DR

Dream2Act introduces a paradigm shift in humanoid robotics: instead of trying to map human motions onto robots (and failing due to skeletal mismatches), it uses generative video models to "hallucinate" the robot itself performing the task. By operating strictly in the robot's native space, it achieves zero-shot interaction success in complex, spatially-sensitive tasks where traditional methods fail completely.

Background: The Retargeting Trap

The "holy grail" of embodied AI is a humanoid robot that can learn from the vast amount of human video data available online. However, the morphology gap remains a fundamental roadblock. Humans and robots have different limb lengths, joint limits, and degrees of freedom. When we use models like SMPL to extract human poses and "retarget" them to a robot, the error accumulates—especially during walking—leading to "phantom" contacts or catastrophic misses.

Motivation: Why Robot-Centric Synthesis?

The authors observe that state-of-the-art video models (like Seedance 2.0) have internalized a "visual physics" through large-scale pretraining. If these models can imagine a human walking, they can also imagine a specific robot—like the Unitree G1—interacting with its environment. By prompting the model with a robot-centric view, the resulting "dream" is already spatially aligned and kinematically plausible for that specific hardware.

Methodology: From "Dream" to Action

The Dream2Act pipeline is a sophisticated three-stage process that bridges the gap between pixels and physical torque.

1. Interaction Hallucination

The system takes a single third-person image of the robot and a target object. It uses Seedance 2.0 to generate a video of the robot completing a semantic prompt (e.g., "sit on the sofa"). Because the model understands the G1's structure, the synthesized video is "morphology-consistent."

2. Native Kinetic Recovery

To extract usable data from the video, the authors built a high-fidelity perception suite:

• Hybrid Dataset: They combined over 5 million frames of simulated G1 data with "Human-in-the-Loop" real-world annotations to train a robust 2D pose estimator.
• 2D-to-3D Lifting: A specialized network regresses 3D joint coordinates from 2D pixel space, explicitly modeling the G1's unique joint distribution.

3. Morphology-Aware Execution

The final stage uses URDF-constrained Inverse Kinematics (IK). By fixing the root and optimizing joint angles within physical limits, they ensure the motion is executable. The global pose is recovered via Perspective-n-Point (PnP), and the whole sequence is fed into SONIC, a general-purpose whole-body tracker.

Experiments: Superior Zero-Shot Performance

The authors tested the G1 on four challenging tasks: Kicking a ball, hugging a box, punching a bag, and sitting on a sofa.

• The 0% vs. 37.5% Gap: Traditional retargeting baselines achieved 0% success because the morphology gap caused the robot to consistently miss the target. Dream2Act achieved an overall success rate of 37.5%.
• Locomotion Robustness: As the distance to the target increased, retargeting errors exploded (up to 0.79m error), while Dream2Act's error remained bounded and stable (~0.14m).

Critical Insights: The Future of Hallucinated Planning

Dream2Act proves that we don't necessarily need to "solve" human-to-robot retargeting to leverage visual data. Instead, we can use generative models as high-level planners.

Limitations: The primary bottleneck is currently inference latency. Since it relies on diffusion models, this is an offline planner. Real-world, reactive adjustments are still handled by the low-level controller (SONIC). However, as diffusion models become faster (via distillation or specialized hardware), we can imagine a future where robots "re-dream" their plans in real-time as they encounter new obstacles.

Takeaway

By shifting the frame of reference from the human to the robot during the generation phase, Dream2Act bypasses a decade of kinematic mapping hurdles, paving the way for truly scalable, data-driven humanoid intelligence.

Detailed Comparisons

In free-space tasks (Text-to-Motion), traditional pipelines often hallucinate "floating" robots or motions that violate balance. Dream2Act's reliance on a video-world model ensures that gravity and ground-contact constraints are implicitly respected, leading to much more stable deployments.