What has to be solved before VLM-guided exploration for vision-language-action policies works outside controlled demos?

Why do VLA policies fumble in the real world? The spatial reasoning gap.

The first bottleneck is that vision-language models (VLMs) inherit weak 3D spatial understanding from their training on 2D images and text. When you ask a VLA policy to 'pick up the red cup on the left,' it may understand the words but misjudge depth, occlusion, or exact coordinates. DepthVLA [1] directly addresses this by adding a pretrained depth prediction module that feeds spatial information into the model. In real-world tasks, this lifted success from 65.0% to 78.5% — a 13.5-point jump that shows how much of the gap is purely spatial. The same paper found that relying on extensive action-data pretraining to 'ground' the VLM in 3D is inefficient and still insufficient. So the fix isn't just more data; it's explicitly injecting depth awareness into the architecture.

This spatial gap is also why simply repurposing a VLM for action prediction underperforms. CogACT [2] shows that a componentized architecture with a specialized action module — using diffusion transformers for action sequence modeling — outperforms a direct VLM adaptation by over 35% in average success rate on simulated tasks and 55% in real robot experiments, despite having the same 7B model size. The lesson: the action head matters as much as the language understanding. A VLM can reason about the scene, but it needs a dedicated, well-designed module to translate that reasoning into precise motor commands.

What happens when the robot gets stuck? The recovery and reward problem.

Even with good spatial reasoning, VLA policies are brittle when they encounter off-nominal states — a cup knocked over, an object out of view, a gripper that missed. ReCoVLA [3] tackles this by keeping the pretrained VLA frozen and using an external VLM to infer the failure mode and recovery stage, then compiling a structured reward for training a residual recovery policy. In simulation, this reward compiler improved average success from 36.7% for a fine-tuned baseline to 66.7% — nearly doubling performance. In real-world zero-shot sim-to-real transfer, it achieved 61.7% success. The key insight is that you don't need the VLM to generate actions directly; you need it to understand what went wrong and guide the recovery process.

The reward problem is even more fundamental. Real-world reinforcement learning with VLA models is bottlenecked by sparse, handcrafted rewards — the robot only gets a reward when it completes the whole task, which is rare and unhelpful for learning. VLAC [4] solves this with a general process reward model that outputs a dense progress signal (how much closer you are to the goal) and a done signal, eliminating task-specific reward engineering. In real-world manipulation tasks, this lifted success from about 30% to about 90% within 200 interaction episodes, and adding human-in-the-loop guidance improved sample efficiency by 50% and reached up to 100% success. This shows that dense, semantically meaningful rewards are essential for efficient learning outside controlled demos.

Can we just combine these fixes? The integration challenge.

The papers suggest that no single fix is sufficient; the real challenge is integrating spatial awareness, robust action generation, and reward understanding into one coherent system. DepthVLA [1] uses a mixture-of-transformers design that unifies a VLM, a depth transformer, and an action expert with fully shared attention — an attempt to combine perception and action in one model. CogACT [2] emphasizes the action module's scaling behavior, showing that diffusion transformers scale favorably with model size, which is promising for future improvements. ReCoVLA [3] and VLAC [4] both decouple high-level understanding from low-level control, but in different ways: ReCoVLA uses a VLM as a reward selector, while VLAC uses a single model that alternates between generating rewards and actions. This diversity suggests that the field is still converging on the best architecture.

There's also a cautionary note from a different domain: VLM-guided exploration isn't just a robotics problem. VLM-Fuzz [5] applies a similar idea to Android GUI testing, where a VLM helps reason about complex screens on-demand, but it still relies on a heuristic-based depth-first search for standard interactions. It outperformed baselines by 9% in class coverage, but the point is that VLM guidance is most effective when used strategically, not as a blanket replacement for structured exploration. This mirrors the robotics findings: you need a strong base policy (or heuristic) and use the VLM to handle the hard cases. So the answer to 'what has to be solved' is not one thing but a combination: spatial reasoning, failure recovery, reward design, and careful integration of these components into a system that can generalize beyond the demo environment.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2024 to 2026, 5 from 2024 or later, collectively cited 540 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 58 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions