Mamoda2.5: Scaling Unified Video Generation and Editing with DiT-MoE
Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE
Mamoda2.5 is a unified Autoregressive-Diffusion framework from ByteDance that integrates multimodal understanding, generation, and editing. It features a 25B-parameter fine-grained Diffusion Transformer with Mixture-of-Experts (DiT-MoE) that achieves SOTA video editing quality, matching proprietary models like Kling O1 while activating only 3B parameters.
TL;DR
Mamoda2.5 is a 25B-parameter unified model from ByteDance that masters multimodal understanding, image/video generation, and instruction-based editing. By replacing standard dense layers with a fine-grained Mixture-of-Experts (MoE) architecture (128 experts), it only activates 3B parameters per pass, achieving unprecedented efficiency. It sets new records in video editing quality, outperforming proprietary leaders like Kling O1, and utilizes a novel joint Distillation-RL framework to deliver 4-step video editing at nearly 100x the speed of current open-source baselines.
Background & Motivation: The "Compute Wall" of Video
As video generation moves toward cinematic quality (Sora, HunyuanVideo), we face a "compute wall." Spatiotemporal tokens grow exponentially with resolution and duration, making traditional dense Transformers (DiTs) too slow for practical use.
The ByteDance team identified three gaps in the current landscape:
- Sparsity Gap: While MoE is popular in LLMs, its application in fine-grained token-level routing for video DiTs is under-explored.
- Data Gap: High-quality "before/after" video editing pairs are rare.
- Latency Gap: Multi-step denoising with Classifier-Free Guidance (CFG) is too slow for real-world creative tools.
Methodology: The Fine-Grained MoE Backbone
The heart of Mamoda2.5 is its DiT-MoE block. Unlike coarse MoEs that route based on the entire image or timestep, Mamoda2.5 uses:
- 128 Routed Experts with Top-8 Routing: This creates over potential expert combinations per token, allowing for extreme specialization.
- Sigmoid Gating & Expert Bias: Instead of Softmax, Sigmoid allows experts to be assessed independently. "Expert Bias" prevents "hot" experts from being over-utilized without needing a complex auxiliary loss.
- In-Context Conditioning: Instead of Cross-Attention, all signals (visual latents, instructions, text) are concatenated. This allows for deeper, bidirectional fusion in the latent space.
Figure 1: The unified AR–Diffusion pipeline of Mamoda2.5.
The Upcycling Breakthrough
Starting an MoE from scratch is expensive. The authors propose a Dense-to-MoE Upcycling procedure. Because their experts (1,024 neurons) are much narrower than the original dense FFN (14,336 neurons), they use Random Neuron Sampling. This preserves the knowledge of the dense model while immediately breaking expert symmetry, leading to a 5x faster training convergence.
SOTA Video Editing & Performance
Mamoda2.5 shines in complex editing tasks like Add, Remove, and Replace.
Benchmark Victories
On OpenVE-Bench, Mamoda2.5 achieved an overall score of 3.86, notably higher than Kling O1 (3.69) and other top-tier proprietary models. It shows significant advantages in spatial reasoning, such as placing an added backpack on the correct shoulder of a moving subject, where other models fail.
Figure 2: Performance comparison across major video generation and editing benchmarks.
15x Acceleration: Joint Distillation & RL
The most impressive practical feat is the 4-step distilled model. The team combined Distribution Matching Distillation (DMD) with DiffusionNFT (RL).
- The Result: A CFG-free student model that requires only 4 steps.
- The Surprise: The RL-enhanced student actually surpasses the 30-step teacher in quality. In some cases where the 30-step model fails to add objects correctly, the 4-step RL model succeeds.
Figure 3: Qualitative comparison on the "Add" task against Kling O1.
Critical Insight & Conclusion
Mamoda2.5 proves that unified models do not have to compromise. By leveraging sparse MoE, the model maintains the "perceptual brain" of a VLM while gaining the "creative hand" of a DiT.
The 98% success rate in internal advertising production suggests that we are moving toward a future where AI handles not just the creation of content, but also its moderation and restoration in a single, efficient loop. The next frontier? Integrating audio for a fully "Omni" audio-video experience.
Limitations
Despite its success, the model relies on a heavy VLM (Qwen3-VL-8B) as a condition encoder, which could still be a memory bottleneck compared to lightweight text-only variants, though the in-context conditioning mitigate fusion errors.