[CVPR 2024] DreamVideo-Omni: Breaking the Motion-Identity Trade-off in Video Customization

TL;DR

DreamVideo-Omni is the first unified framework capable of controlling multiple human/object identities alongside omni-motion (global paths, local limb movements, and camera angles). By introducing a two-stage training process—supervised fine-tuning with specialized structural embeddings and a first-of-its-kind Latent Identity Reinforcement Learning—it ensures that characters don't "lose their face" even during high-intensity action sequences.

The Motivation: Why Video Personalization is Broken

In the current landscape of AI video, you usually have to choose one:

1. Identity Preservation: The character looks right, but stays mostly static (DreamBooth-style).
2. Motion Control: The movement is precise (ControlNet/Trajectory-style), but the character's face melts or transforms into someone else during the movement.

This "Identity Degradation" happens because standard diffusion losses incentivize pixel-wise reconstruction, which conflicts with the pixel-variation required by motion. Furthermore, when two characters are on screen, the model often gets "confused" about which trajectory belongs to which person.

Methodology: The Architecture of Control

1. Stage One: Omni-Motion SFT

The authors adapt a Video Diffusion Transformer (DiT) and solve the "ambiguity" problem through three surgical architectural changes:

• Condition-aware 3D RoPE: Instead of standard positional encodings, they use specialized temporal indices to tell the model: "This token is a static reference, and this token is a moving frame."
• Hierarchical Motion Injection: Rather than just feeding bounding boxes at the start, they inject them into every transformer block, ensuring the spatial geometry remains "locked" throughout the denoising process.
• Group & Role Embeddings: This acts as a "ID Tag." It explicitly binds a specific reference image to a specific trajectory, preventing the "cross-contamination" of movements between subjects.

2. Stage Two: Latent Identity Reinforcement Learning (LIReFL)

This is the "special sauce." Traditional reinforcement learning (RL) in video is slow because you have to decode latents back to pixels to check if the person looks right.

DreamVideo-Omni introduces the Latent Identity Reward Model (LIRM). It operates entirely in the latent space, using a pre-trained Video Diffusion backbone to provide motion-aware identity rewards. It penalizes "copy-paste" artifacts (where the face stays too frozen) and rewards "perceptual consistency" (where the identity stays recognizable through a 3D turn).

Experiments: Small Model, Big Performance

Despite being built on the Wan2.1-1.3B backbone, DreamVideo-Omni crushes much larger models.

The qualitative results are striking: when a subject moves behind an object or performs sudden turns, DreamVideo-Omni maintains high-frequency facial details that other models lose to motion blur or "identity drift."

Critical Analysis & Takeaways

The brilliance of DreamVideo-Omni lies in its efficiency. By moving Reward Learning into the latent space, the authors have made it possible to align video models with human preferences without needing a supercomputing cluster for every fine-tuning run.

Limitations: Like most DiT-based models, "Reward Hacking" remains a risk. If the reward weight (${\lambda }_2$) is too high, the model might produce hyper-realistic faces that don't match the motion's physics.

Future Impact: This work sets a new bar for the "DreamOmni Bench," challenging the community to stop looking at single-subject static videos and start tackling complex, multi-entity storytelling.