[CVPR 2025] WHOLE: Reconstructing 4D Hand-Object Interactions in the Wild
WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos
WHOLE is a world-grounded reconstruction framework that jointly estimates 3D hand articulation and 6D object trajectories from egocentric videos. By leveraging a diffusion-based generative motion prior guided by visual observations and VLM-derived contact cues, it achieves SOTA performance on the HOT3D dataset.
TL;DR
Reconstructing how we manipulate objects from a first-person perspective is notoriously difficult due to motion blur and constant occlusions. WHOLE (World-Grounded Hand-Object Lifted from Egocentric Videos) addresses this by treating reconstruction not as a simple detection problem, but as a guided generative process. By learning what "plausible" movement looks like through a diffusion prior, it can "fill in the blanks" when an object leaves the frame or is hidden by a hand.
The Problem: Perception in the Blind Spot
When you move a box from a table to a shelf while wearing a camera, two things happen that break standard CV algorithms:
- Egocentric Chaos: The camera moves violently, making static objects appear as if they have high velocity.
- The Interaction Paradox: To estimate the hand, you need to know where the object is (occlusion), and to estimate the object, you need the hand (contact). Doing them separately leads to "floating" objects or hands passing through solid geometry.
Methodology: Joint Generative Reconstruction
WHOLE shifts the paradigm from "detect-then-track" to "generate-under-constraints."
1. The Generative Prior
The core is a Transformer-based Diffusion Model. Unlike prior models that only look at 2D pixels, this model is trained on a "gravity-aware" local frame. It learns the mutual dynamics: if a hand moves upward while in contact with a bottle, the bottle must follow.
2. VLM-Guided Logic
How do we tell the generative model which specific video it’s looking at? The authors use Classifier Guidance. They project the generated 3D poses back into 2D and compare them against masks. Critically, they use a Vision-Language Model (VLM) to act as a "contact sensor." By overlaying masks and asking the VLM (like GPT-4o or Gemini) "Is the hand touching obj_1?", they get high-level semantic constraints that physics-based losses often miss.
Experiments: Superior Robustness
The model was tested on the HOT3D dataset. The most impressive results come from the "Out-of-view" and "Truncated" categories.
- Hand Pose: WHOLE achieves a WA-MPJPE of 3.26, outperforming existing world-space hand trackers.
- Object Pose: In cases where objects were partially out of view, WHOLE maintained an ADD-S of 43.8%, whereas the previous SOTA (FoundationPose) collapsed to 19.6%.
- Temporal Stability: Because it uses a motion prior, the resulting trajectories are incredibly smooth (as seen in the ACC-NORM metrics) compared to jittery per-frame estimators.
Critical Insight: Beyond Reconstruction
The authors demonstrate a fascinating application: HOI Planning. Since the model is generative, you can give it a vague hand trajectory and a "grasp" command, and it will synthesize a variety of physically plausible ways to pick up an object. This bridges the gap between Computer Vision (understanding what happened) and Robotics (planning what could happen).
Conclusion
WHOLE proves that for complex human-centric tasks, probabilistic generation is a stronger backbone than deterministic regression. By embedding physical "common sense" into a diffusion prior and guiding it with multimodal VLMs, we can finally reconstruct the 3D world with the same continuity as human perception.
Limitations to Watch: The model currently assumes a known object template and processes pairs independently. Future iterations that handle unknown objects or complex multi-object scenes will be the final frontier for egocentric understanding.