[CVPR 2024] UltraDexGrasp: Breaking the Data Bottleneck in Bimanual Dexterous Grasping

TL;DR

UltraDexGrasp is a breakthrough framework designed to give bimanual robots the "universal" grasping capabilities of humans. By generating a massive dataset of 20 million frames (UltraDexGrasp-20M) through a hybrid optimization-and-planning pipeline, the authors trained a policy that can handle everything from tiny 3g screws to massive 1kg boxes. It achieves an 81.2% success rate in the real world using zero-shot sim-to-real transfer, significantly outperforming existing diffusion and optimization baselines.

Problem & Motivation: The "Strategy Gap"

While single-hand dexterous grasping has seen significant progress, bimanual (two-handed) grasping remains a "frontier" problem. Humans naturally adapt their strategy:

• Small objects: Two-finger pinch or three-finger tripod.
• Medium objects: Single-hand wrap (whole-hand grasp).
• Large/Heavy objects: Coordinated bimanual lifting.

Current robotic systems usually fail because they lack the data to learn these transitions. Real-world teleoperation is too slow to scale, and Reinforcement Learning (RL) often produces "brittle" experts that only learn one way to grab one specific object. UltraDexGrasp asks: Can we synthesize a diverse enough dataset to teach a robot to choose the right strategy automatically?

Methodology: Synergy of Optimization and Planning

The framework operates in two distinct phases: Synthesis and Demonstration.

1. Grasp Synthesis via Bilevel Optimization

Instead of random sampling, the authors use a nonlinear bilevel program.

• Lower Level: Optimizes contact forces to ensure the grasp can resist external forces (wrench space).
• Upper Level: Adjusts the hand pose (translation, rotation, joint angles) to minimize energy terms like hand-object collision and inter-hand penetration.

Fig. 1: The data generation pipeline. The synthesizer handles the "where to put the fingers," while the planner handles "how to get there."

2. Closed-Loop Policy Architecture

The policy is designed for robustness. It takes a raw point cloud from the scene, processes it through a PointNet++ encoder to extract geometric features, and uses a Transformer with unidirectional attention to predict the next action. Crucially, it predicts a bounded Gaussian distribution of actions rather than a single vector, which the authors found improves training stability by 10%.

Fig. 2: The neural architecture. Note the use of action query tokens that attend to scene features to generate control commands.

Experiments: Dominating the Benchmark

The researchers tested the policy on 600 objects in simulation and 25 diverse objects in the real world.

Simulation Performance

The policy achieved an 84.0% average success rate. One of the most impressive findings was the scaling law: as the training data increased from 1M to 20M frames, the success rate on unseen objects continued to climb, proving the dataset's diversity is key to generalization.

Zero-Shot Sim-to-Real

To bridge the "reality gap," the team used:

• Imaged Point Clouds: Adding the robot's own simulated model back into the point cloud to clean up noisy real-world sensor data.
• Impedance Randomization: Varying the "stiffness" of the robot joints during training so the policy doesn't rely on perfect physics.

Table 1: Success rates across different object sizes. The proposed method (Ours) maintains high performance where single-hand methods like DexGraspNet fail (Large objects).

Critical Insights & Conclusion

UltraDexGrasp demonstrates that Multi-Strategy is the future of general-purpose robotics. By moving away from "one-size-fits-all" grasping and embracing a framework that understands when to use two fingers vs. two hands, we move closer to human-level dexterity.

Future Outlook: While the success rate is high, the system currently treats grasping as a standalone task. The next logical step would be integrating this into Functional Grasping—where the robot doesn't just pick up a hammer but picks it up in a way that allows it to actually use it.

Project Page: https://yangsizhe.github.io/ultradexgrasp/