[ECCV 2024] ABMamba: Breaking the Transformer Bottleneck for Efficient Video Captioning

TL;DR

ABMamba is a "fully open" Multimodal Large Language Model that shifts the paradigm from Attention to State Space Models (SSMs). By introducing a novel Aligned Hierarchical Bidirectional Scan (AHBS), it handles long video sequences with linear complexity, offering 3x the inference throughput of Transformer-based models while maintaining or exceeding SOTA accuracy on major benchmarks like VATEX.

Problem & Motivation: The Attention Tax

The leap from image understanding to video understanding is computationally expensive. Standard Transformers use Attention mechanisms where the cost grows quadratically ($O(L^2)$) with the sequence length. In a video context—where you have dozens of frames and hundreds of visual patches per frame—this "Attention Tax" forces researchers to downsample videos aggressively, losing the very temporal nuances (like a specific gesture or a quick action) that define the scene.

The authors of ABMamba identify a critical gap: existing MLLMs are either proprietary (closed-weights) or computationally inefficient. They ask: Can we build a model that understands complex temporal dynamics without the quadratic overhead?

Methodology: The Aligned Hierarchical Bidirectional Scan (AHBS)

The core innovation is the AHBS module, which acts as the bridge between the frozen vision encoders (SigLIP and DINOv2) and the Mamba language backbone.

1. Dual Vision Encoding

ABMamba uses a "best of both worlds" approach by concatenating features from SigLIP (strong semantic alignment) and DINOv2 (fine-grained geometric detail). This ensures the model "sees" both the object categories and the subtle textures.

2. Multi-Resolution Hierarchical Scanning

Unlike standard Mamba scans that process a sequence in a single line, AHBS processes the video through three parallel pathways at different temporal strides:

• Level 1: Full resolution (Captures rapid changes).
• Level 2 & 3: Downsampled resolutions (Captures long-range context).

Each pathway performs a Bidirectional Scan, meaning it looks at the frames both forward and backward in time, ensuring that the causal nature of the Mamba backbone is compensated for by a global temporal awareness.

Experiments & Results: Speed Meets Accuracy

ABMamba was tested on VATEX and MSR-VTT. The results were striking when comparing performance-to-size ratios.

Quantifiable Superiority

• Decoding Speed: ABMamba clocked 95.4 tokens/second, nearly 3x faster than Video-ChatGPT (38.1 tokens/s) and Video-LLaVA (28.9 tokens/s).
• Memory Footprint: During inference, ABMamba only required an increase of 482 MB of VRAM, whereas some Transformer baselines peaked at over 22,000 MB for similar tasks.
• Accuracy: On VATEX, its BLEU4 score of 28.6 outperformed the 7B-parameter LLaVA-OneVision (17.6) and Video-LLaVA (24.5).

Visual Reasoning Case Study

In qualitative tests, ABMamba correctly identified a man using a squeegee on a window, whereas competitors like InternVL2.5 hallucinated a "spray bottle." This suggests that the AHBS module effectively preserves the visual tokens necessary for precise action identification.

Critical Insight: Why Does It Work?

The secret sauce is the sub-quadratic scaling. By utilizing Mamba’s selective scan mechanism, the model doesn't need to compute an $NimesN$ matrix for every frame. Instead, it maintains a compressed internal "state" that it updates as it moves through the video. The Hierarchical design ensures that even as the state compresses data, it doesn't lose the "high-frequency" details of the video (like a fast-moving ax in the VATEX dataset) because one of the scan pathways is always operating at full resolution.

Conclusion & Future Outlook

ABMamba is a significant milestone for Open-Source AI. It demonstrates that we can surpass the efficiency of "standard" Transformer architectures by rethinking sequence modeling.

Limitations: The authors noted that "Object Hallucination" remains the primary error mode (68 out of 100 examined failures). This suggests that while the temporal modeling is solved, the cross-modal alignment between visual and language tokens still has room for improvement—perhaps through "Early Fusion" techniques in future iterations.

For developers in robotics, edge computing, or streaming analytics, ABMamba provides a blueprint for real-time video understanding that doesn't require a cluster of H100s to achieve 30 frames per second.