[CVPR 2025] FluxMem: Smarter, Faster Streaming Video Understanding via Adaptive Hierarchical Memory

TL;DR

FluxMem is a training-free, plug-and-play framework that allows Multimodal Large Language Models (MLLMs) to process infinite video streams with unprecedented efficiency. By implementing a hierarchical memory system that "forgets" redundant spatial and temporal data through Otsu-based adaptive thresholding, it slashes latency by ~70% while actually improving accuracy across major benchmarks like StreamingBench and MLVU.

The Problem: The "Memory Wall" in Streaming Video

As we move from offline video QA (where the whole file is available) to streaming video (robotics, smart glasses, AR), MLLMs face a paradox. If they keep every visual token, the KV cache explodes, leading to massive latency and "Out of Memory" errors. If they prune tokens aggressively using fixed ratios, they often lose the "needle in the haystack"—that tiny spatial detail or brief temporal movement critical for answering a query.

Existing SOTA methods like TimeChat-Online use global pruning rules. However, video is inherently non-uniform. A static security camera feed needs almost 99% compression, while a fast-paced drone racing video requires dense token preservation. FluxMem's core insight is that compression must be data-driven and hierarchical.

Methodology: How FluxMem "Thinks" About Memory

FluxMem organizes visual data into three tiers, much like human sensory, short-term, and long-term memory:

1. Short-term Memory ($M_s$): Retains raw tokens for immediate grounding (the "now").
2. Mid-term Memory ($M_m$): Uses Temporal Adjacency Selection (TAS). It compares a frame with its neighbors. If a token hasn't changed much (low semantic motion), it's dropped.
3. Long-term Memory ($M_l$): Uses Spatial Domain Consolidation (SDC). It groups remaining tokens into "anchors" by merging spatially adjacent regions that are semantically similar.

The Secret Sauce: Otsu’s Adaptive Thresholding

Instead of a human engineer guessing "we should keep 50% of tokens," FluxMem uses Otsu’s Method—a classic image processing algorithm—to analyze the distribution of token similarities in real-time. It automatically finds the "sweet spot" to separate informative tokens from noise.

Figure 1: The FluxMem workflow—cascading tokens from raw input to compressed long-term anchors.

Experiments & Results

The authors tested FluxMem on the Qwen2.5-VL-7B backbone. The results are striking because they defy the typical accuracy-efficiency trade-off.

• Online Prowess: On StreamingBench, FluxMem achieved 76.4, surpasing proprietary models and training-based competitors.
• Efficiency Gains: On OVO-Bench, it achieved a 69.9% reduction in latency and a 34.5% reduction in peak GPU memory usage.
• Offline Generalization: Even in "batch" mode (MLVU), FluxMem outperformed the baseline by 5.2 points while using 65% fewer tokens, proving that filtering out "visual noise" actually helps the LLM focus on salient information.

Table 1: FluxMem vs. SOTA. Note the significant lead over training-based models like VideoLLM-Online.

Critical Insight: Why Does Pruning Increase Accuracy?

Counter-intuitively, FluxMem often performs better than the "full-token" baseline. This suggests a Signal-to-Noise Ratio (SNR) benefit. By removing repetitive background tokens and temporal redundancy, the LLM's limited context window is filled with higher-density semantic information. It no longer has to "wade through" thousands of identical tokens representing a static wall or a still floor.

Conclusion & Future Outlook

FluxMem represents a shift toward autonomous memory management for AI. By making the framework training-free and adaptive, the authors have provided a blueprint for deploying massive MLLMs on edge devices with limited memory.

Limitations: While TAS and SDC are lightweight, they still rely on cosine similarities in the feature space. In extremely low-light or "noisy" sensor environments, the adaptive thresholding might need further robustification to distinguish between sensor noise and actual scene motion.

Future Work: Integrating FluxMem with asynchronous proactive output (where the model decides to speak before being asked) could be the next frontier for truly "live" AI assistants.