[ICRA 2025/2026] EgoAVFlow: Why Robots Shouldn't Just "Look" Like Humans
EgoAVFlow: Robot Policy Learning with Active Vision from Human Egocentric Videos via 3D Flow
EgoAVFlow is a novel framework for learning robot manipulation and active viewpoint control solely from egocentric human videos. It utilizes a shared 3D flow representation to bridge the human-robot embodiment gap and introduces a visibility-maximizing diffusion-based policy that adaptively adjusts camera poses during task execution.
TL;DR
EgoAVFlow breaks the convention of passive robot perception. Instead of naively mimicking a human's head movements from egocentric videos, it learns to actively adjust its viewpoint to keep objects visible. By representing the world through 3D Flow, it bridges the gap between human and robot bodies, achieving nearly double the success rate of previous SOTA methods without requiring a single frame of robot-specific training data.
Problem: The "Human Prior" Trap
Egocentric videos (like those from GoPro or AR glasses) are goldmines for robot learning. However, they come with a hidden curse: Human-specific priors.
- Saccades: Human eyes move rapidly and inconsistently.
- VOR (Vestibulo-Ocular Reflex): Our heads move to stabilize gaze in ways a robot arm-mounted camera doesn't need to.
- Occlusion Blindness: Humans "know" where an object is even if it's partially blocked; a learned policy often does not.
When robots passively imitate these human viewpoints, they often lose track of the object due to self-occlusion or workspace limits.
Methodology: Mastery of the 3D Flow
EgoAVFlow introduces a tri-modular architecture based on Diffusion Transformers (DiT). The "Secret Sauce" is the 3D Flow representation.
1. The Shared Interface
The system tracks pixels in 2D and unprojects them into 3D space using depth data. This creates a "flow" that represents motion independent of what the agent (human or robot) looks like.
2. Visibility-Aware Planning
Instead of just "sampling" a camera pose, EgoAVFlow uses Soft Value-Based Denoising.
- Predict the Future: It forecasts how the object will move (Future Flow) based on planned robot actions.
- Raycasting for Reward: It "imagines" a camera trajectory and uses a physics-based raycaster to check if any obstacle (or the robot's own arm) blocks the view.
- Refine: It biases the diffusion process toward the "visible" trajectory.
Figure 1: The EgoAVFlow pipeline—Predicting actions, flows, and optimized viewpoints simultaneously.
Experiments: Robots in the Wild
The researchers tested EgoAVFlow on four challenging tasks. The setup involved a "duet" of robots: one Trossen WidowX for manipulation and a Unitree Z1 for carrying the camera (Active Vision).
Key Comparison: HVI vs. EgoAVFlow
Human Viewpoint Imitation (HVI) fails when the human's head movement doesn't provide a clear line of sight for the robot's specific kinematics. EgoAVFlow, by contrast, "deviates" from the human path if it means getting a better look at the target.
Table 1: Success rates across tasks. EgoAVFlow achieves up to 2.5x the performance of 2D-based methods like AMPLIFY.
Critical Insight: Why 3D Flow Wins
Most previous methods (like Phantom or AMPLIFY) rely on 2D image features or synthesized robot appearances. These are highly sensitive to viewpoint changes. Because EgoAVFlow reasons in 3D metric space, the policy becomes view-invariant. Whether the camera is 10cm to the left or 20cm higher, the 3D flow of the "spray bottle" remains a consistent mathematical signal for the policy to follow.
Conclusion & Future Work
EgoAVFlow proves that Active Vision isn't just a "nice-to-have"—it's a fundamental requirement for transferring skills from humans to robots.
Limitations: The current system requires target points to be visible at the start of the task. It cannot yet "search" for a lost object from scratch. The Takeaway: Future robot foundations shouldn't just learn to act; they must learn to look—optimizing their own perception to ensure high-stakes tasks don't fail due to a simple case of "blindness."