[CoRL 2024] HoMMI: Scalable Whole-Body Mobile Manipulation via Robot-Free Human Demonstrations

TL;DR

HoMMI (Whole-Body Mobile Manipulation Interface) is a breakthrough framework that enables robots to learn "whole-body" skills—combining navigation, bimanual coordination, and active vision—directly from humans using iPhones, without ever needing a physical robot for data collection. By bridging the "embodiment gap" through 3D geometry and relaxed head-control representations, it transforms a standard bimanual mobile robot into a capable assistant capable of navigating large rooms to do laundry or deliver boxes.

Academic Positioning: This work is an "evolutionary leap" for the UMI (Universal Manipulation Interface) lineage. While prior UMI works focused on local manipulation, HoMMI tackles the much harder problem of global context and cinematic coordination.

Problem & Motivation: The Gap Between Hands and Head

Existing data collection for robots is either high-fidelity but slow (Teleoperation) or fast but nearsighted (Handheld UMI). Teleoperation requires expensive hardware for every hour of data, while UMI uses simple handheld grippers but suffers from "tunnel vision" because it only sees through the wrist cameras.

If we simply slap a GoPro on a human's head to provide global context, we face two massive "Embodiment Gaps":

1. The Visual Gap: A robot's head is usually at a different height, and its cameras see the metal arms of the robot, whereas the human data sees human skin and clothing.
2. The Kinematic Gap: Humans have flexible necks. If the robot tries to mimic every twitch of a human's 6-DoF head pose using its limited 2-DoF neck, the whole system becomes unstable or breaks.

Methodology: Bridging the Human-Robot Divide

The researchers introduced three key innovations to make human data "digestible" for robots:

1. Embodiment-Agnostic 3D Vision

Instead of feeding raw pixels (which contain human arms), HoMMI lifts 2D images into a 3D pointmap. By transforming everything into a "gripper-centric" coordinate frame and masking out everything "behind" the grippers, the policy becomes "blind" to whether the arm is made of flesh or metal.

2. The "3D Look-at Point"

Instead of teaching the robot exactly how to turn its head (6-DoF), HoMMI teaches the robot where to look. The policy predicts a 3D point in space—a Look-at Point. The robot’s internal controller then figures out the best way to orient its head to see that point, effectively transferring the "intent" of active perception (searching for a target) without the kinematic headache.

Figure 1: The HoMMI System Overview - From iPhone data collection to 3D representation and whole-body execution.

3. Constraint-Aware Whole-Body Control

The researchers implemented a differential Inverse Kinematics (IK) solver using Mink. This controller ensures that while the arms are reaching for a mat, the robot doesn't tip over (Center-of-Mass support) or collide with itself, all while keeping the torso upright.

Experiments & Results: Real-World Versatility

HoMMI was tested on three challenging, long-horizon tasks:

• Laundry: Picking up cloth and searching for a bin (Navigation + Active Vision).
• Delivery: Carrying a box across a 6x6m room to a trolley (Large-scale Navigation).
• Tablescape: Unfolding a mat bimanually (Precision Coordination).

Figure 2: Success rates across different tasks. HoMMI (Ours) consistently outperforms baselines that lack either global context (Wrist-Only) or 3D representations (RGB-Only).

Key Finding: Naive RGB-only head sensing failed 100% of the time in the Laundry task because the "viewpoint shift" from human to robot was too high. HoMMI’s 3D approach maintained a 90% success rate.

Critical Analysis & Conclusion

HoMMI demonstrates that the bottleneck in mobile manipulation isn't just "more data," but the right representation of that data.

Takeaway: By treating the head as a search tool (Look-at point) rather than a rigid sensor, we unlock the ability to use massive amounts of "wild" human video/demonstrations.

Limitations:

• Latency: The system involves a complex stack of iPhones and workstations which adds inference lag.
• Tactile Deficit: The system is vision-only; tasks requiring delicate force feedback (like clicking a small button) are still out of reach.

Future Outlook: This framework paves the way for "Foundation Models" for mobile robots, where the data collection interface is so cheap that anyone with three iPhones can contribute to a global robot-instruction dataset.