Why average success rate alone is a misleading yardstick
Most VLA papers report average success on a handful of hand-crafted scenes, but that number hides how fragile the policy is when the world changes. A fuzzing study that generated varied robotic manipulation scenes for seven VLA models found that performance degrades significantly with changes in lighting, camera pose, the number of confounding objects, and even small mutations to the task instruction [3]. In plain terms: a model that nails a fixed lab setup can fail badly when a shadow falls or a camera shifts, so a fair evaluation must systematically vary these factors and report success under each condition, not just a single average.
The same study also tested unseen objects and found that current VLA models lack the robustness needed for real deployment [3]. That means a fair benchmark should include novel objects and scene configurations, not just the ones seen during training. Without this, a high average success rate can be an artifact of overfitting to the evaluation scenes.
Does the model actually reason, or is it pattern-matching?
A fair evaluation should probe whether the policy uses intermediate reasoning or just maps pixels to actions. One approach, CoT-VLA, explicitly predicts future image frames as visual goals before generating actions, and this explicit reasoning improved real-world manipulation success by 17% over a state-of-the-art VLA baseline and by 6% in simulation [4]. That suggests that measuring not just the final action but the quality of intermediate predictions (e.g., whether the predicted future frames are accurate and useful) can reveal whether the model is truly planning.
However, not all reasoning is equally valuable. The same study notes that current VLAs mostly learn direct input–output mappings, lacking temporal planning [4]. So a fair evaluation should include tasks that require multi-step reasoning and should measure whether the model's intermediate outputs (like predicted goals) are coherent and lead to successful actions. Simply measuring final success might miss cases where the model 'gets lucky' without genuine understanding.
Speed and data efficiency are part of real-world usefulness
A policy that succeeds but is too slow to control a robot in real time is not deployable. One study found that a compact VLA, TinyVLA, significantly outperformed the state-of-the-art OpenVLA in inference speed and data efficiency while achieving comparable or better success [5]. In contrast, another study showed that fine-tuning OpenVLA with an optimized recipe (parallel decoding, action chunking, continuous actions, and L1 regression) boosted average success on the LIBERO benchmark from 76.5% to 97.1% while increasing action generation throughput by 26× [1]. These numbers show that evaluation must report both success and latency, because a model that is 20% slower might be unusable for high-frequency control.
Data efficiency is equally important: TinyVLA eliminated the need for a large pre-training stage, which is a major practical advantage [5]. A fair evaluation should therefore measure how much robot data is required to reach a given success level, not just the final success after extensive training. This is especially relevant for real-world deployment where collecting robot demonstrations is expensive.
What a complete evaluation should include
Pulling the evidence together, a fair evaluation of VLM-guided exploration for VLA policies should include at least four components: (1) success rate across systematically varied scenes (lighting, camera, object counts, unseen objects, instruction mutations) [3]; (2) a measure of reasoning quality, such as the accuracy of predicted future frames or intermediate goals [4]; (3) inference speed and data efficiency, reported alongside success [1][5]; and (4) real-world validation on a physical robot, not just simulation, because simulation benchmarks can overstate performance [1][4].
The studies also highlight a tension: optimizing for speed (e.g., TinyVLA) may trade off some performance, while optimizing for success (e.g., OpenVLA-OFT) may require more compute. A fair evaluation should therefore present a Pareto front—showing the trade-off between success and speed—rather than a single ranking. This is the only way to fairly compare models with different design priorities.
About These Sources
This answer is built on 5 peer-reviewed studies — published in 2025, 5 from 2024 or later, 1 in Q1–Q2 journals, collectively cited 73 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 49 papers retrieved from a database of over 500 million.
Sources used in this answer
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
Fine-tuning OpenVLA with an optimized recipe (parallel decoding, action chunking, continuous actions, L1 regression) raised average success on LIBERO from 76.5% to 97.1% and increased action generation throughput by 26×, and outperformed other VLAs and imitation learning baselines on real-world bimanual tasks by up to 15% absolute.
CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving
The CoVLA dataset provides over 80 hours of real driving videos with paired trajectories and language descriptions, enabling MLLMs to generate coherent language and action outputs for autonomous driving, but the abstract does not report quantitative success rates.
VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation
A fuzzing framework (VLATest) tested seven VLA models and found that performance degrades significantly with changes in lighting, camera pose, number of confounding objects, unseen objects, and task instruction mutations, indicating a lack of robustness for practical deployment.
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
CoT-VLA, which predicts future image frames as visual goals before generating actions, outperformed a state-of-the-art VLA by 17% in real-world manipulation and 6% in simulation, demonstrating the value of explicit visual chain-of-thought reasoning.
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation
TinyVLA, a compact VLA with a diffusion policy decoder, significantly outperformed OpenVLA in inference speed and data efficiency while achieving comparable or superior success, and showed strong generalization across language instructions, novel objects, positions, appearances, backgrounds, and environmental shifts.