Why visual generalization is the first thing to break in production
The strongest evidence across these studies points to visual out-of-domain (OOD) scenarios as the primary failure mode. A 2025 study of three open vision-language-action models found that none exhibited robustness to visual OOD scenarios, and specifically showed that OpenVLA—which uses two pre-trained vision models—suffered catastrophic forgetting in DINO-v2, failing depth regression [3]. This means that even small changes in lighting, background, or object appearance can cause a production robot to misidentify or fail to perceive its environment, leading to task failure.
The same study showed that recovering this visual generalization ability improved grasping and lifting performance in OOD tasks by 77% and 66% respectively [3]. That's a massive jump, indicating that visual robustness is not a nice-to-have but a core requirement for production. Another study, using Octo in an industrial simulation, found significant performance degradation despite minimal task and observation domain shifts, attributing this to limitations in visual generalization and language grounding [5]. Together, these findings suggest that if you're deploying a robot foundation model, you should expect visual OOD failures first and plan for them.
Language grounding and safety: the silent killers in production
Beyond vision, language grounding—the model's ability to correctly map natural language instructions to actions—is another critical failure mode. The Octo study explicitly identified language grounding limitations under distribution shifts as a key issue [5]. In production, this means a robot might misunderstand a slightly rephrased command or fail to follow a novel instruction, leading to incorrect actions. For example, Diffusion-VLA, a different model, showed the ability to follow novel instructions, but this was in a controlled setting; the gap between best-case and typical-case evidence is stark [1].
Safety is another major concern. A 2026 paper proposed a framework to enforce formal safety specifications (like time-bounded goals and persistent safety conditions) during execution of a pretrained foundation model, because these models are data-driven and lack formal guarantees [4]. This is a direct acknowledgment that without explicit safety constraints, production robots could violate operational rules, which is unacceptable in real-world environments. The framework modifies the action distribution at each step to satisfy safety constraints, but it's an add-on, not a built-in property—highlighting that safety is a separate failure mode that must be actively managed.
The gap between best-case and typical-case evidence
The papers here show a clear divide: some models perform impressively in controlled settings, but degrade sharply in production-like conditions. Diffusion-VLA, for instance, achieved 63.7% accuracy on zero-shot bin-picking of 102 unseen objects and ran at 82Hz on a single GPU [1]. That's a strong best-case result. But the same paper notes that the model needed less than 50 demonstrations for a complex task, which is a far cry from the massive datasets needed for robust generalization. The Octo study, on the other hand, showed that even with minimal domain shift, performance degrades significantly in simulation [5].
This gap is not just about model architecture; it's about the evaluation environment. The Octo study used a simulation-first approach to adapt real-world-trained models to synthetic environments, which is a critical step for safe industrial deployment [5]. The ReVLA study also emphasized that visual OOD failures are caused by limited training data variation and catastrophic forgetting [3]. So, while a model might shine in a benchmark, production environments introduce variability that these models are not trained to handle. The takeaway: don't trust best-case numbers; test your model under the exact conditions you'll deploy in, and expect failures in visual and language domains first.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2024 to 2026, 5 from 2024 or later, 1 in Q1 journals, collectively cited 63 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 43 papers retrieved from a database of over 500 million.
Sources used in this answer
Diffusion-VLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression
Diffusion-VLA, a framework combining autoregression and diffusion, achieved 63.7% accuracy on zero-shot bin-picking of 102 unseen objects and ran at 82Hz on a single A6000 GPU, but also showed that it could train on less than 50 demonstrations for a complex task, indicating data efficiency but also potential limitations in generalization.
Robot learning in the era of foundation models: a survey
A 2025 survey of foundation models in robot learning identified critical issues including robot hardware and software decoupling, dynamic data, and generalization performance in the presence of humans, and called for future research on multimodal interaction and robotics-specific foundation models.
ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models
A 2025 study of three open vision-language-action models found that none were robust to visual out-of-domain scenarios, and OpenVLA specifically suffered catastrophic forgetting in DINO-v2, failing depth regression; a gradual backbone reversal approach improved grasping and lifting in OOD tasks by 77% and 66% respectively.
Specification-Aware Distribution Shaping for Robotics Foundation Models
A 2026 paper proposed a specification-aware action distribution optimization framework that enforces Signal Temporal Logic constraints during execution of a pretrained robotics foundation model without modifying its parameters, addressing the lack of formal safety guarantees in data-driven models.
Beyond performance: Explaining generalisation failures of Robotic Foundation Models in industrial simulation
A 2025 study using Octo in an industrial simulation found significant performance degradation despite minimal task and observation domain shifts, and identified limitations in visual generalization and language grounding under distribution shifts, along with architectural and benchmarking challenges.