ShareVerse: Breakthrough in Multi-Agent Consistent Video Generation for Shared World Modeling

TL;DR

ShareVerse is an innovative framework that transforms large-scale video models into "Shared World Models." Unlike previous works that focus on isolated agents, ShareVerse allows multiple independent agents to explore the same environment while maintaining strict global consistency. By integrating Cross-Agent Attention and a 4-view spatial concatenation strategy, it ensures that what one agent sees is perfectly synchronized with another agent's perspective, even during dynamic interactions.

Academic Positioning: This work represents a significant leap from Single-Agent Trajectory Prediction to Multi-Agent Collaborative Generation, filling a critical gap in the simulation of shared physical realities for embodied AI and gaming.

The Challenge: Building a World with No "Single" Point of View

Traditional world models treat the environment as a private canvas for a single entity. When you introduce a second agent, current models struggle with two things:

1. Shared Consistency: If Agent A looks left and sees a red building, Agent B looking right at the same spot must see the same building.
2. Dynamic Perception: Agents must be able to "see" and render each other accurately as they move through the world.

The authors observed that existing datasets and architectures (like SV4D or SyncCamMaster) were too small-scale or object-centric to handle the complexity of urban driving or multi-robot interaction.

Methodology: Connecting Minds through Cross-Agent Attention

ShareVerse addresses these challenges through a three-pronged architectural approach:

1. Spatial Concatenation for 360° Awareness

To give each agent a complete understanding of its surroundings, ShareVerse concatenates four views (Front, Rear, Left, Right) into a single wide-format video stream. This forces the model's internal self-attention to learn the geometric relationship between viewpoints of a single agent.

2. Raymap Encoding for Precise Control

Instead of using raw camera coordinates, the model converts camera intrinsics and poses into Raymaps (representing ray direction and origin). This translates geometric data into a format that the Diffusion Transformer (DiT) can easily process as an element-wise addition to latent features.

3. The Cross-Agent Attention Block

This is the "secret sauce." The framework concatenates the video features of different agents along the frame dimension and passes them through a shared attention block.

• Mechanism: It uses Rotary Position Embeddings (RoPE) to keep track of timing and identity.
• Intuition: By allowing the features of Agent 1 to attend to the features of Agent 2, the model effectively "shares" the visual memory of the world. If Agent 1 has already seen a specific intersection, the model uses that information to generate a consistent view for Agent 2.

Experiments & Results: Seeing the Same World

The model was trained on a custom-built CARLA-based dataset featuring 55,000 pairs of interactive videos across various weather conditions and urban scenes.

Qualitative Excellence

As shown in the visualizations, the model doesn't just generate a background; it perceives the other agent. When two vehicles meet at a junction, both agents' videos correctly render the other vehicle with high temporal and spatial fidelity.

Quantitative SOTA

The model maintains impressive scores across human-centric benchmarks:

• Motion Smoothness: 0.9745 (indicating minimal jitter even with complex camera movement).
• Background Consistency: 0.9312 (ensuring the world doesn't "morph" as agents move).

Critical Insight & Conclusion

The real value of ShareVerse lies in its scalability. By relying on Cross-Agent Attention rather than explicit 3D reconstruction (like NeRF or Gaussian Splatting), it stays within the efficient latent space of Diffusion Transformers.

Limitations: Currently, it is optimized for two-agent interactions. Scaling this to a "swarm" of 10+ agents would require more complex sparse attention mechanisms to avoid the quadratic memory explosion of long sequences.

In summary, ShareVerse successfully bridges the gap between video generation and physical world simulation, providing a robust foundation for the next generation of multi-agent embodied AI.