[CVPR 2025] UniVBench: The "Turing Test" for Unified Video Foundation Models

TL;DR

As the industry moves from specialized video models to Unified Video Foundation Models (VFMs) that both "see" and "create," our evaluation tools have remained stuck in the past. UniVBench breaks this bottleneck by introducing a unified, multi-shot benchmark and an agentic evaluation system (UniV-Eval) that assesses models across Understanding, Generation, Editing, and Reconstruction.

The Perception-Generation Gap

Historically, the AI community has treated video understanding (V2T) and video generation (T2V) as church and state. Understanding models like VideoLLaVA excel at perception but are "blind" creators. Generation models like Sora or Kling produce stunning visuals but cannot reason about the content they synthesize.

The new wave of Unified Models (e.g., Emu3, Show-o, Chameleon) attempts to merge these worlds. However, testing them accurately is nearly impossible with current benchmarks because:

1. Fragmented Metrics: You can't compare a BLEU score (text) with an FVD score (distribution).
2. Single-Shot Bias: Most benchmarks use 5-second clips, failing to test the temporal consistency required for cinematic storytelling.
3. Data Contamination: Models are being tested on the same YouTube clips they were trained on.

Methodology: UniVBench and the V2V Challenge

UniVBench solves this by providing 200 high-quality, human-validated, multi-shot videos.

The Reconstruction (V2V) Insight

The most innovative contribution is the Video-to-Video Reconstruction task. To pass, a model must:

• Phase 1 (Perception): Watch a video and write a dense, detailed caption.
• Phase 2 (Generation): Use its own caption to recreate the original video.

If the reconstructed video fails, UniV-Eval can pinpoint exactly where the chain broke: did the model fail to see the "German Shepherd tilting its head," or did it fail to generate the action it just described?

Figure 1: The UniVBench framework covering 8 cinematic dimensions and 6 unified tasks.

UniV-Eval: The Agentic Judge

UniVBench moves away from "single scalar scores." Using UniV-Eval, the system segments videos into shots and evaluates them against 21 sub-dimensions (Lighting, Subject, Camera Angle, etc.) using a Vision-Language Model as a judge. This provides a "weakness checklist" rather than a cryptic number.

Figure 2: The UniV-Eval workflow—decomposing, planning, and scoring via an agentic loop.

Experimental Insights: The "Unification Gap"

The results from UniVBench are a wake-up call for the industry. The benchmark reveals a massive Unification Gap:

• Perception Leaders: Gemini 2.5 Pro dominates understanding but cannot generate.
• Generation Leaders: Seedance-1.0-Pro leads in T2V but lacks deep reasoning.
• Unified Models: Current unified architectures like Showo-2 or OmniVideo show promising versatility but significantly lag behind specialized models in their respective domains.

Figure 3: Detailed performance breakdown across tasks and cinematic dimensions.

Critical Analysis & Conclusion

UniVBench's greatest value lies in its copyright-free, multi-shot nature. By moving away from web-scraped data, it provides a "clean room" for evaluation.

Limitations: With only 200 videos, it is an evaluation set, not a training set. The community now needs to scale this high-quality annotation density to thousands of videos to train truly unified models.

The Takeaway: UniVBench proves that "Action" and "Temporal Reasoning" remain the Achilles' heel of all current models. As we move toward 2026, the goal is no longer just "generating a pretty video" but building systems that truly understand the cinematic language they are asked to create.