[CVPR/ICRA 2025] MoDE-VLA: Achieving Human-Like Bimanual Dexterity via RL-Augmented Shared Autonomy

TL;DR

MoDE-VLA tackles the "holy grail" of robotics: human-like bimanual dexterous manipulation. By introducing a hierarchical structure that pairs a high-level Vision-Language-Action (VLA) model with RL-trained low-level primitives (IMCopilot), and a Mixture-of-Experts module for force/tactile fusion, the authors achieved the first autonomous dual-hand apple peeling—a task requiring extreme coordination and contact awareness.

Problem & Motivation: The 63-DoF Headache

Current SOTA models like OpenVLA or π0 are great at picking up blocks, but they struggle when the task requires "in-hand" manipulation. Rotating an apple while peeling it requires managing 63 Degrees of Freedom (DoF). For a human operator, teleoperating this via an exoskeleton is cognitively exhausting, often leading to poor data quality.

Furthermore, vision alone isn't enough for "contact-rich" tasks like plugging in a charger or assembling gears. You need to feel the resistance. However, simply "dumping" force data into an LLM/VLM backbone often confuses the model, as it doesn't understand the different temporal scales and physical meanings of torque vs. pixels.

Methodology: The Hierarchical Approach

The framework rests on two pillars designed to solve the data and fusion bottlenecks:

1. IMCopilot (Shared Autonomy)

During data collection, the operator uses an exoskeleton for arm movements but delegates the "hard part" (rotating the object in the fingers) to an RL-trained copilot triggered by a foot pedal. This Shared Autonomy ensures the collected data is high-quality and physically stable. During autonomous execution, the VLA acts as the "brain," deciding when to call the IMCopilot "sub-routine."

2. MoDE-VLA (Mixture-of-Dexterous-Experts)

Instead of adding force/tactile data as just more text tokens, the authors created a dedicated pathway.

• Token Construction: Arm torques and fingertip tactile readings are projected into the embedding space and replicated across the action horizon.
• Sparse MoE Routing: A set of expert MLPs specializes in different regimes (e.g., "contact-onset expert" vs. "stable-grasp expert").
• Residual Injection: The corrections from these experts are added back to the main VLA output. If there’s no contact, the correction is zero, keeping the original VLA's "common sense" intact.

Figure: The MoDE-VLA architecture showing the separate Force/Tactile tokens and the MoE routing mechanism.

Experiments: Beyond Pick-and-Place

The authors tested the system on four escalating tasks: Gear Assembling, Charger Plugging, Tube Rearranging, and the ultimate challenge: Apple Peeling.

Key Results:

• Success Rate: MoDE-VLA doubled the success rate of the π0 baseline (34% vs 15% average).
• Apple Peeling: While the baseline failed completely (0% success), MoDE-VLA achieved a 30% success rate and a 73% completion ratio.
• Ablation Insight: Removing "Force" input caused the biggest drop in performance, proving that for dexterous tasks, "touch" is more important than "sight" during the final millimeters of a task.

Figure: Sequence of the four evaluation tasks. Note the bimanual coordination in the Apple Peeling task.

Critical Analysis & Conclusion

Takeaway

The success of MoDE-VLA demonstrates that we shouldn't expect a single "End-to-End" transformer to learn everything from scratch. Dividing the labor between VLA (High-level planning) and RL (Low-level reactive control) is a much more bio-plausible and efficient path toward humanoid robotics.

Limitations

Despite the breakthrough, a 30% success rate on apple peeling shows we are still far from industrial-grade reliability. The reliance on sim-to-real for IMCopilot also suggests that as we add more skills, the simulation overhead might become a bottleneck.

Future Work

The next step is likely the integration of audio feedback (the sound of the blade on the skin) and scaling the "Skill Repertoire" so the VLA can handle a wider variety of tools beyond peelers and chargers.