VideoRouter: Breaking the Long-Video Bottleneck with Query-Adaptive Dual Routing

TL;DR

The computational cost of processing long videos in Multimodal Large Language Models (MLLMs) is exploding. VideoRouter solves this by treating token compression not as a fixed pruning task, but as a budgeted evidence allocation problem. By using a dual-router system to sense what a question actually needs, it reduces visual tokens by up to 67.9% while actually improving accuracy on benchmarks like VideoMME and LongVideoBench.

The Scalability Wall in Video MLLMs

As we push MLLMs like InternVL or LLaVA to understand hour-long videos, we hit a wall: the number of visual tokens grows linearly with video length, leading to OOM (Out-of-Memory) errors and massive inference latency.

Current solutions usually fall into two traps:

1. Static Compression: They apply the same "pooling" or "dropping" rule to every frame, ignoring that some frames are vital and others are filler.
2. Weak Query Awareness: They decide which tokens to keep before the model has even "read" the question properly.

The authors of VideoRouter observed that long-video evidence is highly non-uniform. A "What color is the car at 05:00?" query needs high-resolution detail at a specific timestamp, whereas "Summarize the movie" needs broad temporal coverage at low resolution.

Methodology: The Dual-Router Architecture

VideoRouter reuses the early layers of the LLM (the "shadow stack") to perform lightweight routing before the full heavy inference begins.

1. The Semantic Router (The "What Strategy?" Phase)

The Semantic Router analyzes the text query to pick an allocation policy:

• Global Policy: Best for holistic questions. It applies uniform compression across the whole video.
• Fragment Policy: Best for localized questions. It triggers the Image Router to find specific "evidence frames."

2. The Image Router (The "Where is the Evidence?" Phase)

When in Fragment mode, the Image Router scores every frame's relevance to the question. It identifies "Critical Frames" (high fidelity) and "Background Frames" (aggressive compression).

3. Priority-Aware Pooling

The final sequence is reconstructed based on a strict budget (e.g., fitting everything into 12k tokens). If the budget is tight, it discards background frames first, ensuring that vital evidence—the "Critical Frames"—remains untouched.

Specialized Supervision: Training the Decision Makers

You can't train a router without knowing what "relevant" looks like. The authors created two massive datasets:

• Video-FLR-200K: 200k pairs with frame-level relevance labels, found by identifying the "minimum sufficient frames" to answer a question.
• Video-QTR-10K: 10k pairs labeled with the best allocation strategy (Global vs. Fragment).

Experimental Performance

The results prove that "smarter" is better than "denser."

• Accuracy vs. Efficiency: VideoRouter (Ours) achieves 66.5% on Video-MME Overall, outperforming the InternVL3 baseline while using significantly fewer tokens.
• Cross-Backbone Success: When adapted to Qwen2.5-VL, VideoRouter improved LongVideoBench scores by 4.4 points and reduced TTFT (latency) simultaneously.
• Ablation Studies: Removing the Semantic Router (policy-level) or Image Router (frame-level) consistently degraded performance, proving the necessity of the dual-routing approach.

Critical Insights & Takeaways

1. The Pareto Frontier: The study found that reusing the first 4 layers of the LLM for routing strikes the perfect balance between routing accuracy (83.4%) and latency overhead (~6%).
2. Context Window Management: By calculating the $B_{img}$ (Visual Budget) dynamically ($L_{max}-L_{text}-L_{gen}$), VideoRouter ensures zero OOM errors, making it highly robust for production deployments.
3. Inductive Bias: The work suggests that for long-video understanding, the model's ability to select information is just as important as its ability to process it.

Conclusion

VideoRouter represents a shift in thinking for MLLM efficiency. Instead of engineering faster kernels or smaller weights, it optimizes the information flow. By sensing query intent and scanning for visual evidence, it allows current models to "see" further into long videos without losing sight of the details that matter.

Limitations: The current method assumes offline access to video frames for routing and was primarily validated on InternVL/Qwen architectures. Future work may explore online streaming routing for real-time applications.