UniVidX: Redefining Video Diffusion as a Unified Multimodal Engine

TL;DR

UniVidX is a breakthrough framework that transforms Video Diffusion Models (VDMs) from simple "text-to-video" generators into versatile multimodal engines. By introducing dynamic masking and decoupled adaptation, it masters 15 diverse tasks—including video matting, relighting, and inverse rendering—within a unified architecture. Remarkably, it achieves SOTA results even when trained on tiny datasets (<1k videos), proving the immense power of latent diffusion priors.

The Paradigm Shift: From Fixed Mappings to Omni-Directional Generation

Traditionally, if you wanted to perform Video Matting, you trained a matting model. If you wanted Normal Estimation, you trained a perception model. This "one-model-one-task" philosophy has two major flaws:

1. Inflexibility: Models are locked into rigid input-output roles.
2. Inconsistency: Modalities like Albedo, Normals, and RGB are physically coupled, but separate models treat them as independent, leading to flickering and misalignment.

The authors of UniVidX asked a bold question: Can we let a single model decide which modalities are "conditions" and which are "targets" on the fly?

Methodology: The Three Pillars of UniVidX

The architecture, built on the Wan2.1-T2V-14B backbone, relies on three core innovations:

1. Stochastic Condition Masking (SCM)

Instead of hard-coding the model to take Text and output RGB, SCM randomly partitions modalities during training. In one step, the model might see RGB and predict Normals; in the next, it might see a Text prompt and predict RGBA layers. This forces the model to learn the joint distribution of all visual modalities.

2. Decoupled Gated LoRA (DGL)

Sharing weights across modalities with vastly different distributions (e.g., Albedo vs. Depth) usually leads to "feature interference" and structural collapse.

• Decoupled: Each modality gets its own LoRA.
• Gated: These LoRAs are only active when a modality is a target (noised). When a modality is a condition (clean), the model uses the frozen, original VDM weights to extract features, preserving the "native" knowledge of the 14B model.

3. Cross-Modal Self-Attention (CMSA)

To ensure that a generated Albedo map actually matches the generated RGB video, UniVidX shares Keys and Values across all modalities in the attention layers. This allows a pixel in the "Normal" stream to "look at" the corresponding pixel in the "RGB" stream, ensuring perfect spatial and temporal alignment.

Figure 1: The UniVidX framework utilizes SCM, DGL, and CMSA to enable versatile generation.

Exceptional Data Efficiency

One of the most striking findings of the paper is its data efficiency.

• UniVid-Intrinsic was trained on only 900 synthetic clips.
• UniVid-Alpha was trained on just 484 clips.

Despite this, UniVidX beats specialized models trained on hundreds of thousands of frames. For example, in Normal Estimation, it outperformed NormalCrafter while using 45x less training data. This suggests that the 14B parameter backbone already "understands" physics and geometry; it just needs a lightweight "steering" mechanism to output those modalities.

Results & Applications

The framework's versatility enables complex downstream workflows:

• Video Relighting: Extract Albedo/Normals $\to$ Change Text Prompt $\to$ Re-render RGB with new lighting.
• Background/Foreground Replacement: Decompose video into layers $\to$ Swap background prompt $\to$ Re-composite.

Table 1: Quantitative comparison showing UniVidX outperforming specialized baselines across multiple metrics.

Critical Analysis & Outlook

While UniVidX is a major leap forward, it faces typical "large model" hurdles. The 14B backbone is memory-intensive, limiting current outputs to 480p and 21 frames. Furthermore, the model can inherit biases from its small task-specific datasets—for instance, failing to calculate the transparency (alpha) of an ice cube because the training data was focused on human portraits.

However, the core takeaway is clear: Unified multimodal modeling is the future. Instead of building a "Swiss Army Knife" of separate tiny models, we are moving toward "Generalist Engines" that treat every vision task as a conditional generation problem.

Summary for Researchers: UniVidX proves that parameter-efficient fine-tuning (LoRA) combined with smart modality masking is sufficient to repurpose T2V models for high-precision graphics tasks.