[CVPR 2024] ART-STVG: Breaking the 60-Second Barrier in Spatio-Temporal Video Grounding

TL;DR

Existing Spatio-Temporal Video Grounding (STVG) models are "short-sighted," typically failing when videos exceed one minute due to memory explosions and informational noise. ART-STVG (AutoRegressive Transformer) redefines the task as a streaming process. By using selective memory banks and a cascaded spatial-to-temporal decoder, it achieves a massive +9.1% m_tIoU gain on 3-minute videos while slashing GPU memory usage by over 60%.

Context & Motivation: The "All-at-Once" Bottleneck

In the world of video understanding, STVG is the "holy grail"—find where and when an object exists based on a text query (e.g., "The man in the red shirt kicks the ball"). However, current benchmarks like HCSTVG and VidSTG only average 20-30 seconds.

When you push these models to 5-minute videos, two things happen:

1. Memory Crash: Simultaneous attention across thousands of frames exhausts VRAM.
2. Signal Dilution: The target event is buried under a mountain of irrelevant frames, leading to "localization drift."

The authors argue that we should process video like humans do: sequentially, keeping a "memory" of what matters and ignoring the rest.

Methodology: The ART of Autoregression

ART-STVG abandons the parallel processing paradigm in favor of an AutoRegressive Transformer.

1. The Streaming Architecture

Instead of encoding the whole video, ART-STVG processes frames one by one. It fuses 2D appearance (ResNet) and 3D motion (VidSwin) features with RoBERTa text embeddings in a multimodal encoder.

2. Cascaded Spatio-Temporal Decoder

Unlike previous works that treat "where" (spatial) and "when" (temporal) as parallel tasks, ART-STVG connects them. The spatial decoder first finds the box; then, an RoI Pooling layer extracts motion features specifically from that box to help the temporal decoder decide if the event is starting or ending. This "fine-grained" hint is crucial for long videos.

3. Selective Memory: Filtering the Noise

Storing every frame in memory would eventually lead back to the same bottleneck. ART-STVG uses two selective banks:

• Spatial Selection: Keeps only the top-N memories most similar to the text query.
• Temporal Selection: Uses a "TextTiling" inspired approach to identify event boundaries and only retrieves memories within the "current" event segment.

Experimental Breakthroughs

The team extended HCSTVG-v2 to create the LF-STVG-1min/3min/5min benchmarks.

SOTA Comparison

As shown in the table below, while previous models like TubeDETR and TA-STVG see their performance plummet as video length increases, ART-STVG maintains high accuracy. On 5-minute videos, the m_tIoU gap between ART-STVG and the next best model is a staggering 7.3%.

Efficiency Gains

Perhaps the most impressive result is the resource efficiency. ART-STVG requires only 7.9GB of GPU memory regardless of video length, compared to 25.1GB for TA-STVG on the same batch. This makes long-form grounding accessible on consumer-grade GPUs.

Critical Insight: Why it Works

The success of ART-STVG lies in its Inductive Bias. By forcing the model to look at the video frame-by-frame, the creators introduced a form of temporal consistency. The memory selection acts as a "hard attention" mechanism, preventing the model from getting distracted by a similar-looking object appearing 2 minutes after the actual target event.

Conclusion & Limitations

ART-STVG is a pioneer in the LF-STVG space. It proves that autoregressive structures are not just more efficient but more accurate for long sequences.

Limitations:

• Inference Speed: Sequential processing is slower (1.09s vs 0.69s) than parallel models for shorter clips.
• Drift: If the spatial decoder fails on a key frame, the cascaded design might propagate that error to the temporal head.

For future work, the industry should look into combining this autoregressive logic with faster architectures like State Space Models (SSMs) to achieve real-time long-form grounding.