[CVPR 2026] AVGen-Bench: Exposing the "Semantic Hollow" Behind Cinema-Grade AI Video

TL;DR

As generative AI moves from silent clips to full-blown Text-to-Audio-Video (T2AV) synthesis, our evaluation tools haven't kept pace. AVGen-Bench steps in as a rigorous, task-driven benchmark that ignores "vibe-based" metrics in favor of granular truth. It reveals a startling reality: today's top models produce breathtaking visuals but fail miserably at basic logic—like playing a C-major scale or rendering legible background text.

Background: The Illusion of Perfection

We are entering the era of "Sora-class" models where the visual fidelity is essentially solved. However, a cinematic video of a pianist is useless if the audio plays a random jazz riff while the fingers strike a classical chord. This modality misalignment and semantic fragility are what AVGen-Bench aims to quantify. It moves the goalposts from "Does this look real?" to "Does this do what the user actually asked for?"

Methodology: Hybrid Reasoning for Complex Worlds

The core innovation of AVGen-Bench is its Multi-Granular Evolution Suite. Instead of a single "black box" score, it breaks down evaluation into signal-level precision and high-level reasoning.

1. The Expert-LLM Pipeline

Most benchmarks use CLIP embeddings, which are "blind" to details. AVGen-Bench uses:

• Specialist Models: PaddleOCR for text, InsightFace for identities, and Basic-Pitch to convert audio into symbolic MIDI data.
• MLLM Reasoning: It feeds these structured signals into Gemini (acting as a "Multimodal Auditor") to check against the original prompt's logic.

Figure: The AVGen-Bench framework bifurcates into Signal-level precision (orange) and Semantic reasoning (purple).

2. Task-Driven Curation

The authors curated 235 prompts across three domains:

• Professional Media: Testing narrative beats and multi-shot consistency.
• Creator Economy: Testing ASMR, tutorials, and specific musical chords.
• World Simulation: Using "Underspecified Prompting" to see if the model knows that sodium should explode when dropped in water without being told.

Critical Findings: Where Models Break

The results from testing titans like Sora 2, Veo 3.1, and Kling 2.6 provide a sobering reality check.

The "Musical Pitch" Disaster

One of the most significant "universal breakdowns" found was in musical control. While models generate perfect instrument textures, they have zero understanding of music theory. When prompted for a "C Major scale," every single model generated random frequencies. Figure: The "Semantic-Neural Disconnect"—Visuals show a piano, but the audio MIDI analysis reveals chaotic noise.

The Persistence of "Glyph Collapse"

Text rendering remains a "dual-mode" failure. Models can handle short, explicit titles but collapse into "gibberish scribbles" for incidental text (like a sign in the background). This proves models lack a fundamental "world knowledge" of how text exists in 3D space.

Identity Drift and Physics

Facial consistency still drops as scenes get crowded (Crowd Degradation), and physical plausibility scores remain below the passing threshold. Models are currently "texture generators," not "physics engines."

Deep Insight: Beyond Texture Generation

The takeaway for the AI community is clear: Scaling laws for aesthetics do not automatically solve for semantics.

AVGen-Bench proves that we need new training paradigms. We must move beyond "coarse-grained" alignment where we just match a video to a caption. Instead, we need Fine-Grained Supervision—perhaps by training models on symbolic data (like MIDI or OCR-grounded videos) to ensure the "world model" actually understands the rules of the world it is rendering.

Conclusion

AVGen-Bench is a vital "reality check" for the AIGC industry. It provides the diagnostic tools necessary to move from making "pretty pictures that move" to building reliable, controllable, and physically grounded multimodal intelligence.

Project Link: aka.ms/avgenbench