[CVPR/IROS Style] Pixel2Catch: Rethinking Dynamic Manipulation via Pixel-Level Motion & Heterogeneous MARL

Executive Summary

TL;DR: Pixel2Catch enables a robot arm equipped with a multi-fingered hand to catch objects thrown by humans using only a single RGB camera. By ditching explicit 3D coordinate estimation in favor of pixel-level visual cues and adopting a Heterogeneous multi-agent reinforcement learning (MARL) structure, the system achieves impressive sim-to-real performance without any real-world fine-tuning.

Positioning: This work shifts the paradigm of dynamic manipulation from "Geometry-first" (sensing 3D space) to "Perception-first" (reasoning through visual changes), bridging the gap between human-like visual intuition and robotic execution.

The "3D Trap" in Dynamic Manipulation

Most prior works in robotic catching treat the problem as a two-step process: 1) Estimate the 3D trajectory using Motion Capture (MoCap) or Depth (RGB-D) sensors, and 2) Command the robot to that coordinate.

However, this approach often fails in the real world because:

• Sensor Dependency: Depth sensors are noisy and have limited range.
• Sim-to-Real Gap: Simulation provides "perfect" coordinates, but real-world vision systems introduce latencies and measurement errors that break the control loop.
• Control Complexity: Coordinating a 6-DoF arm with a 16-DoF hand (like the Allegro hand) leads to a massive action space that single-agent RL struggles to explore efficiently.

Methodology: Human-Inspired Perception & Decomposition

1. Pixel-Level Motion Features

Instead of calculating $X,Y,Z$, the authors use SAM 2 to track the object's bounding box in 2D. They extract a 6D feature vector: $z_{pixel}=\{c_x,c_y,\Delta c_x,\Delta c_y,\Delta w,\Delta h\}$

• $(c_x,c_y)$ provides the lateral position.
• $(w,h)$ and their changes ($\Delta w,\Delta h$) provide depth/scale cues. As an object gets closer, it gets larger—this is exactly how humans perceive "Time-to-Contact."

2. Heterogeneous MARL (The Arm vs. The Hand)

The system is split into two specialized agents:

• Arm Agent: Focuses on positioning the palm near the object trajectory.
• Hand Agent: Focuses on the dexterous finger movements required to wrap around the object.

Fig 1. The Pixel2Catch Pipeline: Decoupled control for Arm and Hand using role-specific observations.

This decomposition allows for role-specific reward functions. The hand doesn't need to worry about the arm's reachability; it only cares about joint distances to the object, reducing the conflict during the learning process.

Experimental Results & Insights

The authors conducted rigorous testing against Single-Agent (S-A RL) and ablation models (only center, only scale).

SOTA Comparison in Sim and Real

While "Only-Center" models performed well in the highly structured simulation, they crumbled in the real world. Why? Because they couldn't estimate the approach speed without scale ($w,h$) information.

Fig 2. Learning Curves: Heterogeneous MARL (Proposed) consistently reaches higher success rates faster than single-agent setups.

Why MARL works better here?

1. Observation Decoupling: The Arm agent doesn't need to see finger joint states to position the palm, reducing noise.
2. Credit Assignment: By having separate reward channels, the "failure" of a catch can be specifically attributed to either a bad reach (Arm) or a premature grasp (Hand).

Fig 3. Real-world catching of various geometries (Cube, L-block, Triangle) using only mono-RGB.

Critical Analysis & Conclusion

Takeaway

Pixel2Catch proves that visual dynamics (pixel-level changes) are a sufficient proxy for physical dynamics (3D velocity). By combining this with a modular MARL architecture, we can scale dexterous manipulation to high-DoF systems without the "Curse of Dimensionality" typical of single-agent RL.

Limitations

• Single-Arm Constraints: The robot is stationary and has a limited workspace.
• Object Variety: While it handles some geometries, very thin or transparent objects would likely break the SAM 2 segmentation pipeline.

Future Outlook

The move toward Bimanual catching is the next frontier. Splitting the problem into four agents (2 arms, 2 hands) could further enhance the stability and allow for catching much larger or heavier objects that a single Allegro hand cannot secure.