[arXiv 2026] LVOmniBench: The New Frontier for Long-Form Audio-Visual Intelligence

TL;DR

As Omnimodal Large Language Models (OmniLLMs) evolve, the bottleneck has shifted from "understanding an image" to "reasoning over an hour-long movie." LVOmniBench is a pioneering benchmark that challenges models with 10 to 90-minute videos. It exposes a harsh reality: while proprietary models like Gemini 3 Pro are making headway (65.8% accuracy), open-source models are still largely "deaf" to audio cues, barely outperforming random guessing on complex, long-context reasoning.

The "Short-Video" Trap: Why Current Evaluations Fail

Until now, most multimodal benchmarks (like MovieChat or Video-MME) focused on short clips. In the real world, however, context is built over tens of minutes. Current models often "hallucinate" or rely on unimodal shortcuts—for instance, answering a question about a video's plot using only the dialogue (ASR) or static visual frames.

The authors of LVOmniBench identified that existing datasets lacked the "spatiotemporal and acoustic dynamics" necessary to test if a model really understands how sound and sight intertwine over time.

Methodology: High-Fidelity Manual Curation

To solve this, the team didn't just scrape YouTube; they manually filtered 3,000+ raw videos down to 275 high-quality samples.

• Diversity: Covers 5 primary domains (Entertainment, DIY, Film, etc.) across 21 subcategories.
• Annotation Rigor: 1,014 QA pairs were designed to be impossible to answer using only one modality or common sense.
• Difficulty Scaling: Tasks are stratified into Low, Medium, and High tiers, focusing on everything from "Sound Inference" to "Complex Causal Tracking."

The Performance Gap: Proprietary vs. Open Source

The experimental results are a wake-up call for the open-source community.

| Model Category | Key Model | Accuracy (Avg) | High Difficulty Accuracy | | :--- | :--- | :--- | :--- | | Proprietary | Gemini 3 Pro | 65.8% | 45.0% | | Open-Source | Qwen3-Omni-30B | 35.8% | 28.6% | | Open-Source | Ming-Omni-2.0 | 34.6% | 29.3% |

Critical Insights:

1. The Audio Bottleneck: Open-source models struggle to use raw audio. When provided with Gemini-generated ASR transcripts, their performance jumped by ~7-8%, suggesting their current audio encoders are failing to "read" the sound sequences effectively.
2. Long-Context Decay: As video duration increases, accuracy plummets. This is likely due to "token overflow," where models must drop frames or compress audio so much that fine-grained details (like a specific sound at minute 42) are lost.
3. The "Non-Verbal" Struggle: Models are decent at understanding speech but fail miserably at "Music Perception" and "Counting" events through sound.

Error Analysis: Why Models Fail

The authors categorized 153 failures from Gemini 3 Flash and found:

• Perception Errors (34%): Models often ignore audio when visual cues are present (Modality Bias).
• Localization Errors (19.6%): In a 40-minute video, models simply can't find when something happened.
• Cross-Modal Semantic Gap (19%): Models see a person and hear a sound but fail to realize the person is the one making that specific sound.

Critical Analysis & Future Outlook

LVOmniBench proves that Omnimodal ≠ Multimodal. Being able to process text, image, and audio is not enough; a model must align them across thousands of temporal steps.

The Takeaway: The next generation of OmniLLMs needs more than just bigger context windows. They need efficient token compression that doesn't sacrifice temporal continuity and native audio-visual alignment that moves beyond treating audio as a secondary "captioning" track.

LVOmniBench is a high-bar challenge that will likely define the SOTA race for the next two years. For developers, the message is clear: if your model can't handle a 45-minute cooking show where the sound of sizzling matters as much as the chef's actions, it isn't ready for the real world.