[Video-MME 2024] HiMu: Breaking the Pareto Front in Long-Video QA with Hierarchical Logic Trees

TL;DR

Processing long videos (10 mins to 1 hour) in Vision-Language Models is a balancing act between cost and context. HiMu (Hierarchical Multimodal Frame Selection) is a new training-free framework that skips expensive iterative "agentic" loops. Instead, it uses a single LLM call to turn a question into a logical "instruction tree," evaluates it using fast modality experts (Vision + Audio), and picks the best 16 frames to answer the question. It beats models using 32x more frames while being 10x faster.

The "Blind Spot" of Current Selectors

Why is long-video QA so hard? Most current models use one of two extremes:

1. Similarity-based (Fast but Shallow): They turn the whole question into one vector and compare it to frames. But if a question asks: "What happened after the man mentioned the secret ingredient?", a simple CLIP vector can't distinguish between the audio (mentioning the ingredient) and the visual (the subsequent action).
2. Agent-based (Deep but Expensive): They use an AI agent to "watch" segments, think, and zoom in. This is accurate but requires massive FLOPs and high latency.

HiMu fills this gap by asserting that compositional reasoning can be resolved before the model even looks at the video.

Methodology: The Neuro-Symbolic Blueprint

The core of HiMu is the transformation of a messy natural language query into a structured, executable logic tree.

1. Hierarchical Decomposition

A text-only LLM (like Qwen or GPT) parses the query into a JSON tree.

• Leaf Nodes: Specific "tasks" for specialized experts (e.g., OVD for "black dog", ASR for "chemical reaction").
• Internal Nodes: Logical operators like AND, OR, and temporal ones like SEQ (Sequence) or RightAfter.

2. Multimodal Expert Routing

HiMu is one of the first selectors to treat Audio (ASR for speech, CLAP for sounds) as a first-class citizen alongside Vision (CLIP for scenes, OVD for objects, OCR for text).

Figure: The HiMu Pipeline. Logic parsing leads to expert signal extraction, filtered through fuzzy logic to find the "Satisfaction Curve".

3. Fuzzy Logic Composition

Instead of binary "Yes/No" results, HiMu uses continuous fuzzy logic. This allows the model to handle noise and "near misses" in time. The SEQ operator, for instance, ensures that a frame is only highly rated if the "cause" event happened before it and the "effect" event happened after it.

Experiments: Doing More with Less

HiMu was tested on Video-MME, LongVideoBench, and HERBench-Lite.

Key Results:

• Efficiency: HiMu with 16 frames outperformed global similarity methods (like BOLT) and even specialized iterative search methods (like T*).
• The 4x Efficiency Gain: HiMu at 16 frames achieved better accuracy than Uniform Sampling at 64 frames.
• Universal Plug-and-Play: It improved every LVLM it was paired with (Qwen, InternVL, LLaVA, Gemini) without any fine-tuning.

Figure: Accuracy vs. FLOPs. HiMu (green star) sits high above the previous Pareto front, offering agent-level accuracy at similarity-level speeds.

Why it Works: The Interpretability Edge

One of the unique "PhD-level" insights of this paper is the Heatmap of Logic. Because HiMu uses a deterministic logic tree, we can see exactly why a frame was picked. If the model fails, we can check a heatmap to see if the "Audio" expert missed a word or if the "Object Detector" failed to see the dog. This makes the system far more "debuggable" than typical black-box AI agents.

Figure: Every frame's selection can be traced back to the specific expert (OCR, CLIP, OVD) that triggered it.

Conclusion & Limitations

HiMu represents a shift toward Neuro-Symbolic preprocessing for long-form video. It proves that clever "signal composition" is more important than "raw frame count."

Limitations: It still depends on the initial LLM's ability to parse the tree correctly. If the LLM generates a malformed logic tree, the selection fails. Additionally, while it amortizes costs well (caching visual features), the first-time run for a new video still requires extracting features across five different expert models.

Takeaway: For production-grade Video AI, HiMu suggests we should stop trying to build "smarter eyes" and start building "smarter logic" to decide what the eyes should look at.