MolmoAct2: Mastering Real-World Robotics with Adaptive Depth Reasoning and Open Foundation Models

TL;DR

MolmoAct2 is a breakthrough in the quest for a "Generalist Robot Controller." It moves beyond the limitations of closed, slow, and hardware-specific models by introducing a fully open-source ecosystem. With a new spatial-reasoning backbone (Molmo2-ER), a novel continuous action expert, and an adaptive "thinking" mechanism that only updates depth information for changing parts of a scene, MolmoAct2 sets a new SOTA for real-world deployment on affordable robot platforms.

Background: The Gap in Robotic Deployment

The dream of a single model controlling any robot in any kitchen or warehouse has been hindered by three major walls:

1. Proprietary Opacity: Frontier models like ${\pi }_0$ often hide their data and recipes.
2. The Latency Trap: Reasoning-augmented models (predicting "Chain-of-Thought" or "Depth") are often too slow for smooth, closed-loop control.
3. Hardware Elitism: Many models are optimized for expensive, industrial-grade robots inaccessible to most researchers.

Methodology: The "Specialize-then-Rehearse" Recipe

1. Molmo2-ER: Spatial IQ for Robots

Standard VLMs are great at identifying "a cat," but terrible at estimating "how many centimeters to the left" an object is. Molmo2-ER is specialized for Embodied Reasoning. Through a 3.3M sample corpus, it learns pixel-accurate pointing, ego/exocentric correspondence, and metric distance—outperforming GPT-5 and Gemini-ER on spatial benchmarks.

2. The Per-Layer KV Action Expert

Instead of just using the final hidden state of a VLM to predict actions, MolmoAct2 "grafts" a Flow-Matching Continuous Action Expert onto the backbone.

• How it works: Every layer of the action expert cross-attends directly to the KV cache of the corresponding VLM layer.
• Why it works: This allows the control head to access hierarchical visual-semantic features (from raw textures to high-level intentions) throughout the architecture.

Figure 1: Overview of MolmoAct2's architecture featuring the continuous action expert and real-world deployment scenarios.

3. MolmoAct2-Think: Adaptive Reasoning

The "Think" variant introduces Adaptive Depth Reasoning. Unlike predecessors that re-calculate the entire scene's geometry at every timestep, MolmoAct2-Think compares the current frame with the previous one. It only generates new depth tokens for coordinates where the RGB signal has changed (e.g., a moving arm or a shifted cup). This allows the model to stay "3D-aware" while maintaining a high control frequency.

Experimental Results: SOTA Across the Board

MolmoAct2 was tested on the most extensive suite of benchmarks for an open VLA to date:

• Simulation (LIBERO): Achieved a staggering 98.1% success rate, outperforming ${\pi }_{0.5}$ and NORA-1.5.
• Real-World Zero-Shot: Deployed on DROID and SO-100 (a low-cost robot) out-of-the-box, it maintained high success rates (56.7% for SO-100) despite entirely novel objects and random camera poses.
• Trajectory Quality: On RoboEval, the model didn't just succeed; it produced paths that were 2x shorter and significantly more stable than diffusion-based baselines.

Table 1: Molmo2-ER performance vs. proprietary models like GPT-5 and Gemini-ER 1.5 Thinking.

Efficiency and Speed

One of the most impressive feats is the inference optimization. By using CUDA Graphs and KV-caching, the authors boosted the control rate of MolmoAct2 to 55.79 Hz. Even the reasoning-heavy "Think" variant runs at 12.7 Hz, sufficient for most real-world manipulation tasks.

Figure 2: Control rates after Caching and CUDA Graph optimizations, showing significant speedups for real-time control.

Conclusion and Future Impact

MolmoAct2 proves that openness does not come at the cost of performance. By releasing the weights, the code, and the largest open bimanual dataset (BimanualYAM), the AI2 team is providing the community with a robust foundation for the next generation of physical agents. The introduction of adaptive depth reasoning specifically offers a template for how "System 2" thinking can be integrated into high-speed robotic control.

Key Takeaways for Practitioners:

• Fine-tuning works: MolmoAct2 adapts efficiently to new embodiments with just a few dozen demonstrations.
• Spatial backbones matter: Don't just use a general VLM; use one specialized in embodied reasoning (like Molmo2-ER) to see an immediate 6% jump in success rates.
• Openness fuels progress: The release of the OpenFAST Tokenizer allows anyone to train their own VLAs using a standardized, high-quality action vocabulary.