[NeurIPS 2025] Tango: Taming Visual Signals for Efficient Video Large Language Models
Tango: Taming Visual Signals for Efficient Video Large Language Models
Tango is a training-free token pruning framework for Video Large Language Models (Video LLMs) that integrates a diversity-driven salient token selection strategy with Spatio-temporal Rotary Position Embedding (ST-RoPE). It achieves state-of-the-art efficiency, preserving 98.9% of the performance of LLaVA-OneVision while providing a 1.88x inference speedup by retaining only 10% of video tokens.
TL;DR
Efficiency in Video Large Language Models (Video LLMs) is often a battle against the "Token Tsunami." While token pruning is the standard solution, current methods are often "blind" to the complex spatial and temporal structures of video data. Tango introduces a more intelligent pruning framework that uses diversity-driven selection and Spatio-temporal Rotary Position Embedding (ST-RoPE). It maintains 98.9% of original performance while slashing token counts by 90%, delivering nearly 2x inference speedups.
Background: The Limits of Traditional Pruning
Current Video LLMs use two main paradigms for efficiency:
- Saliency-based (Top-k): Keeping tokens with the highest attention scores.
- Diversity-based (Clustering): Merging similar tokens to reduce redundancy.
However, the authors of Tango discovered that these methods have reached a bottleneck. Traditional Top-k selection is too rigid; it focuses on the "attention peaks" but ignores the "long tail" of semantically diverse regions. Furthermore, direct similarity clustering often results in "fragmented objects" (as seen in the image below), where a single person might be split across multiple noisy clusters, destroying the geometric consistency needed for reasoning.
Methodology: The Tango Framework
Tango refines the pruning process through three integrated modules:
1. Diversity-Driven Salient Selection
Instead of a simple Top-k filter, Tango first expands the candidate set (Top-k̄). It then uses Density Peaks Clustering (DPC-KNN) to group these candidates into semantic regions. By picking the most "attentive" token from each cluster, Tango ensures all visual modes (e.g., subtitles, foreground objects, background context) are represented, preventing the model from becoming obsessed with "attention sinks."
2. ST-RoPE: Preserving the "Where" and "When"
To fix the fragmentation issue in clustering, Tango introduces Spatio-temporal Rotary Position Embedding (ST-RoPE).
- The Intuition: Tokens that are physically far apart in time or space should be less likely to cluster together, even if their features look similar.
- The Mechanism: By integrating 3D coordinates (time, height, width) into a rotation matrix, Tango injects a locality prior. This mathematically penalizes the similarity of distant tokens, ensuring that clusters correspond to coherent, continuous spatio-temporal "blobs" rather than scattered noise.
Experiments & Results
Tango was tested on heavy-hitters like LLaVA-OneVision and Qwen2.5-VL across benchmarks like Video-MME and MVBench.
- Near-Lossless Efficiency: At a 10% token retention ratio, Tango achieves a 1.88x speedup while losing less than 1.1% of performance compared to the full-token model.
- Superior Scalability: As the number of input frames increases, Tango’s performance remains stable, whereas other methods suffer from increased noise or redundancy.
- Compatibility: It can be stacked with intra-LLM pruning (pruning within the Transformer layers) for even greater efficiency gains.
Critical Insight: The "Attention Sink" Problem
A fascinating finding in the paper is the analysis of Attention Sinks in the SigLIP vision encoder. The authors found that certain corner tokens consistently attract massive attention scores regardless of content. Tango mitigates this by masking these "sink tokens" during selection, allowing the model to focus on actual semantic content. This highlights a critical need for deeper investigation into how vision encoders are trained and how their inductive biases affect downstream LLMs.
Conclusion
Tango represents a shift from "brute-force pruning" to "structure-aware reduction." By respecting the multi-modal nature of attention and the geometric continuity of video, it sets a new standard for training-free efficiency in Video LLMs. For developers looking to deploy large-scale video understanding models, Tango offers a plug-and-play solution to triple inference throughput without sacrificing visual intelligence.
Limitations: While Tango excels at structure preservation, compressing extremely crowded or abstract scenes (e.g., e-sports or massive crowds) remains a "final frontier" for zero-shot pruning methods.