Why the best-case performance is not what you'll see in practice
The strongest single finding across these studies is that execution-centric VLMs (Vision-Language-Action models) look impressive in high-level reasoning but fall apart on low-level execution. In EmbodiedBench, which tested 24 leading models across 1,128 tasks, the best model (GPT-4o) scored only 28.9% on average—meaning even the top performer failed more than 70% of the time on tasks that mix navigation, manipulation, and household chores [4]. That's not a fluke: the same study found models excel at high-level semantic tasks but struggle with low-level manipulation, which is exactly where embodied execution happens.
This gap between best-case and typical-case is also visible in failure detection. FAER, a benchmark with 3,323 episodes across 65 tasks, found that nearly all VLMs—including GPT-4o—have a high false-negative rate: they tend to ignore abnormal events like dropping an object during a 'clean room' task [3]. So if you're relying on the VLM itself to notice its own mistakes, you'll be disappointed. The practical takeaway: don't trust a model's self-assessment; build an external diagnostic layer that watches execution and flags failures.
The most effective diagnostic tool is a supervisory agent, not the VLM itself
Instead of asking the VLM to diagnose its own failures, the strongest evidence points to a separate supervisory agent that monitors execution and provides corrective instructions. The IEI-TIA system does exactly this: it watches robotic sorting and palletizing tasks, identifies failure types like off-center grasps or misplaced parts, and tells the robot what to do next. After fine-tuning on paired vision-language data, it achieved 93.7% diagnosis accuracy—a ~38% improvement over general-purpose large models—and boosted task success by 14.0% when its instructions were followed [1]. This is a closed-loop approach: the agent doesn't just detect failures; it corrects them before they cascade.
Why does this work? Because execution-centric VLMs are often 'closed-box'—they generate actions without explaining why. A supervisory agent adds interpretability by explicitly classifying failure types and linking them to corrective actions. The IEI-TIA paper notes that minor errors can rapidly propagate in long-horizon tasks, so early diagnosis is critical [1]. For your team, this means investing in a separate diagnostic model—not just fine-tuning the execution VLM—and using paired vision-language data to train it on failure categories specific to your tasks.
Simulation testbeds let you trace failures to specific actions, not just observe them
A third diagnostic tool is a simulation environment that records everything and allows replay. The Lingjing testbed, built for multi-agent urban tasks, generates 'attribution-ready replays' that link each agent's trajectory and communication to changes in the environment, resource consumption, and evaluation metrics [2]. This lets you pinpoint whether a failure came from a grounding error (misunderstanding a spatial instruction), a long-horizon planning breakdown, or a coordination issue between agents. In their tests, Lingjing exposed persistent bottlenecks in grounding and long-horizon execution, and showed that adding more model capacity gave diminishing returns while heavier workloads reduced success [2].
This is different from a benchmark like FAER, which tells you whether a model can detect a failure, or EmbodiedBench, which scores overall success. A testbed gives you the 'why' behind the failure. For example, if your VLM fails a manipulation task, you can replay the episode to see if it misperceived the object's position or made a wrong action sequence. The key is to design your testbed with attribution in mind—record not just outcomes but also intermediate states, actions, and communications, so you can trace the causal chain.
Don't just measure success—measure failure detection and correction
The studies converge on a crucial point: evaluating only task success hides the most important diagnostic information. FAER specifically evaluates four capabilities: failure detection, categorization, description, and correction [3]. This is more informative than a binary success/fail because it tells you where the model breaks down. For instance, a model might complete a task but only after ignoring a critical failure—which is dangerous in real-world settings. FAER's finding that VLMs have high false-negative rates means they often 'succeed' by ignoring problems, which is a hidden risk.
Similarly, EmbodiedBench's 28.9% average score for GPT-4o shows that even top models are unreliable for low-level execution [4]. But if you only looked at high-level task success, you might miss this. The recommendation: build a diagnostic suite that includes failure-aware metrics, not just task completion. Use benchmarks like FAER to test your VLM's ability to detect and reason about failures, and use simulation testbeds to trace those failures to specific causes. This combination gives you a complete picture of where your model succeeds and fails—and what to fix.
About These Sources
This answer is built on 4 studies (3 peer-reviewed, 1 preprint) — published from 2025 to 2026, 4 from 2024 or later, 1 in Q1 journals — selected as the most relevant from 4 studies that passed quality screening, drawn from 40 papers retrieved from a database of over 500 million.
Sources used in this answer
IEI-TIA: Industrial Embodied Intelligence Trustworthy Interpretable Agent for Robotic Long-horizon and Repetitive Tasks
IEI-TIA, a supervisory agent for industrial sorting/palletizing, achieved 93.7% failure-diagnosis accuracy (a ~38% improvement over general-purpose large models) and improved task success by 14.0% when its corrective instructions were followed, in a study of long-horizon robotic tasks.
Lingjing: A Simulation Testbed for Multi-Agent Embodied Tasks in Open-Ended Cities
Lingjing, a simulation testbed for multi-agent urban tasks, provides attribution-ready replays linking trajectories and communications to environment changes; in tests with 12 VLMs on 9 tasks, it exposed persistent grounding and long-horizon execution bottlenecks, with diminishing returns from added model capacity.
FAER: Benchmarking VLMs for Failure-Aware Embodied Reasoning
FAER, a failure-aware benchmark with 3,323 episodes across 65 tasks, found that nearly all 16 VLMs (including GPT-4o) have high false-negative rates in failure detection, meaning they tend to ignore abnormal events during long-horizon execution.
EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
EmbodiedBench, evaluating 24 MLLMs across 1,128 tasks in four environments, found that models excel at high-level tasks but struggle with low-level manipulation, with the best model (GPT-4o) scoring only 28.9% on average.