[CVPR 2026] VAG: Dual-Stream Video-Action Generation for Scalable Embodied Data Synthesis
VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis
VAG (Video-Action Generation) is a unified dual-stream generative framework based on Flow Matching that simultaneously synthesizes temporally aligned video sequences and robot action trajectories. By achieving state-of-the-art performance in video-action consistency, VAG enables large-scale embodied data synthesis and boosts downstream policy success rates (VLA) by 20% absolute.
TL;DR
The bottleneck for universal robot intelligence is data. VAG (Video-Action Generation) breaks this bottleneck by shifting from "collecting" data to "generating" it. By using a unified Flow Matching framework, VAG generates high-fidelity video and aligned actions simultaneously. It doesn't just "dream" about robots moving; it provides the exact motor signals needed to make those dreams executable, resulting in a +20% success rate boost in downstream real-world robot tasks.
The Alignment Gap: Why World Models Aren't Data Engines (Yet)
Current robot learning relies heavily on human teleoperation—a process that is expensive and impossible to scale to "internet-level" datasets. While modern World Models (like Sora or Cosmos) can generate stunning videos of robots performing tasks, they are "mute" in terms of actions.
Previous attempts to solve this used Two-Stage Pipelines:
- Generate a video.
- Use an Inverse Dynamics Model (IDM) to guess the actions.
The result? Misalignment. If the video generation and action inference don't share a common "brain" during the denoising process, the resulting actions often fail to match the visual reality, leading to "ghosting" actions or trajectories that are physically impossible.
Methodology: The Power of Synchronized Denoising
VAG treats video and action as two views of the same underlying physical process.
1. Unified Flow Matching
Instead of standard Diffusion, VAG adopts Flow Matching, which models the probability path as a straight-line velocity. This leads to smoother optimization and faster inference—critical when generating 10-second sequences of high-dimensional robot data.
2. Dual-Stream Architecture & Adaptive 3D Pooling
The architecture features two parallel branches. The Video Stream (based on Cosmos-Predict2) handles the complex spatiotemporal visual generation. The Action Stream (a modified 1D U-Net) generates the motor commands.
The "secret sauce" is the Adaptive 3D Pooling. In every denoising step, the predicted clean video latent is compressed and passed to the action branch. This ensures that the generated action is always conditioned on the global context of the scene, maintaining a rigorous link between pixels and motors.
Experiments: Transforming Simulation to Reality
VAG was tested on the AgiBot (real-world) and LIBERO (simulation) datasets.
Performance Gains
In quantitative comparisons, VAG significantly outperformed the two-stage "Video + AnyPos" baseline. In simulation, the "Replay Success Rate" (the ability for a robot to actually complete a task by following VAG's generated actions) reached 62%, compared to just 54% for the best two-stage competitor.
Real-world Generalization
The most impressive result came from the VLA pretraining. A policy trained only on a small real-world dataset failed when the color of a bowl changed or its position shifted. However, when pretrained on VAG-synthesized data, the robot demonstrated superior generalization, boosting the success rate from 35% to 55%.
Critical Insight: World Models as Policies
VAG isn't just a data generator; it can function as a World-Action Policy. Because it takes an image and text as input and outputs a synchronized video-action pair, it can plan and execute actions in one go. The qualitative results show VAG handling left-arm, right-arm, and complex bimanual manipulations with high fidelity.
Conclusion & Future Outlook
VAG proves that the future of robotics lies in generative simulation. By aligning the "eyes" (video) and "hands" (actions) of the model in a single unified flow, we can finally move past the teleoperation bottleneck.
Next Steps: The authors note that currently, the video branch doesn't "listen" to the action branch (unidirectional conditioning). Future iterations of VAG will likely feature bidirectional feedback, allowing planned actions to influence the simulated visual future, creating a truly interactive world model.
VAG represents a milestone in the "Data-Centric AI" movement for robotics, proving that synthetic data, when properly aligned, can overcome the limitations of physical world data collection.