Why real-world action data is the scarcest resource
Execution-centric VLMs need to learn not just what objects look like, but how to act on them in the physical world. That requires paired data of visual scenes, language instructions, and the resulting actions—data that is expensive and slow to collect in real environments. A 2025 study on human-robot collaborative assembly notes that embodied intelligence 'requires repetitive training of various functional models' to handle dynamic scenarios, which is a direct symptom of data scarcity [3]. Similarly, a 2026 paper on UAV emergency response points out that complete model training is 'computationally infeasible at the edge due to limited compute resources and training data' [5]. The takeaway: without enough diverse, real-world action examples, models can't learn the fine-grained motor and decision skills needed for tasks like assembly or disaster response.
The problem is worse for novel or rare situations. A 2024 study on CLIP (a vision-language model) found that when fine-tuned only on base classes, it struggles to generalize to novel classes lacking visual samples during training [1]. This is the same issue in execution: if a robot has never seen a particular object or scenario, it can't reliably decide how to act. The study's solution—using synthetic data—helps, but it requires careful alignment with real data to avoid distribution mismatch [1]. So the bottleneck isn't just data volume; it's the coverage of edge cases that real-world deployment demands.
The reward annotation bottleneck and the sim-to-real gap
Reinforcement learning (RL) for embodied agents depends on reward functions that tell the agent what behavior is good. Hand-designing these rewards is a major bottleneck: it's time-consuming, error-prone, and struggles with sparse rewards and cross-domain data scarcity, according to a 2026 survey [2]. The survey reviews five landmark VLM-based reward annotation methods (VIP, LIV, Eureka, DrEureka, EMMA-X) and finds that while they automate reward labeling, they still face challenges like visual grounding errors and high latency [2]. In plain terms: even when you use a VLM to generate rewards, the model might misinterpret what it sees or be too slow to provide real-time feedback, which slows down training and can lead to suboptimal behavior.
Another layer is the sim-to-real gap: models trained in simulation often fail in the real world because the data distributions differ. The 2026 survey explicitly compares methods on 'sim-to-real transferability' and notes it as a key challenge [2]. A 2025 study on digital twin-assisted assembly tries to bridge this by using a digital twin as a simulation engine, but it still requires VLM-enhanced RL with flexible reward design [3]. The bottom line: even with automated reward annotation, the lack of high-quality, real-world data to validate and fine-tune those rewards remains a bottleneck.
Compute and data efficiency: the edge-device squeeze
Embodied systems often run on resource-constrained hardware—like drones or robots—where large pre-trained VLMs are impractical. A 2026 study on UAV emergency response shows that adapting small VLMs through fine-tuning is essential, but complete model training is 'computationally infeasible at the edge' due to limited compute and training data [5]. Their solution is data-efficient fine-tuning, which achieves 91% accuracy on flood detection, but it still requires strategic fine-tuning at edge nodes or monitoring stations, not fully onboard [5]. This highlights a practical bottleneck: even if you have data, you may not have the compute to use it effectively in the field.
Neuromorphic engineering offers a potential path, as a 2022 perspective argues that brain-inspired hardware could enable low-power, real-time processing for embodied robots [4]. However, that approach is still in early stages and faces open challenges [4]. For now, the data bottleneck is compounded by hardware limits: you can't just throw more data at a model if the device can't handle the training or inference load. This is why data-efficient methods—like the fine-tuning strategies in [5]—are critical, but they also mean you need to be selective about what data you use, which further constrains the learning process.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2022 to 2026, 4 from 2024 or later, 3 in Q1 journals, collectively cited 187 times — selected as the most relevant from 7 studies that passed quality screening, drawn from 52 papers retrieved from a database of over 500 million.
Sources used in this answer
Synth-CLIP: Synthetic data make CLIP generalize better in data-limited scenarios
Synth-CLIP shows that synthetic data can improve CLIP's generalization to novel classes in data-limited scenarios, but requires cross-domain feature alignment to match real and synthetic samples; it outperformed PromptSRC by 3.0% on novel classes across 11 datasets.
Vision-Language Models for Automated Reward Annotation in Embodied AI: State-of-the-Art, Challenges, and Future Directions
A 2026 survey of five VLM-based reward annotation methods (VIP, LIV, Eureka, DrEureka, EMMA-X) identifies visual grounding errors and high latency as key challenges, and emphasizes the difficulty of sim-to-real transfer and cross-domain data scarcity.
Vision language model-enhanced embodied intelligence for digital twin-assisted human-robot collaborative assembly
A 2025 study on VLM-enhanced embodied intelligence for human-robot collaborative assembly shows that VLMs can circumvent repetitive training of perception models, but still require reinforcement learning with flexible reward design and digital twin simulation to handle dynamic environments.
Embodied neuromorphic intelligence
A 2022 perspective on embodied neuromorphic intelligence argues that brain-inspired computing could enable low-power, real-time processing for robots, but identifies open challenges and calls for further research.
Towards Practical UAV Intelligence: Data-Efficient Fine-Tuning of Small Vision Language Models for Emergency Response
A 2026 study on UAV emergency response shows that data-efficient fine-tuning of small VLMs achieves 91% accuracy on flood detection, but complete model training is computationally infeasible at the edge due to limited compute and training data.