ReToken: Unlocking Long-Context VLMs with a Single Learnable Token

TL;DR

Processing hour-long videos or massive image collections often breaks Vision-Language Models (VLMs) because they can't effectively "find the needle in the haystack." ReToken solves this by adding a single learnable embedding that performs retrieval in the Value space rather than the traditional Attention (Query-Key) space. It delivers up to a 13.4-point boost on Visual Haystacks and transfers zero-shot to long videos with minimal computational overhead.

The Problem: The Attention Signal is Broken

As visual context grows, VLMs face two hurdles:

1. Memory Bottlenecks: You simply cannot fit thousands of visual tokens into a standard GPU KV cache for a single forward pass.
2. Retrieval Failure: Standard attention scores (Query-Key similarity) are surprisingly poor at identifying relevant frames. The authors found that for a Qwen3VL-8B model, attention-based retrieval achieves a measly 5.1% average Recall@1 on egocentric video tasks.

The fundamental issue is that attention is trained for next-token prediction, not information retrieval. In most training sets, the image is always relevant to the question, so the model never learns to ignore distractors.

The Insight: Value Features are the Key

The core technical contribution of ReToken is the shift from Query-Key scoring to Value-based similarity. In a Transformer, the "Value" (V) represents the actual content being propagated, while Query (Q) and Key (K) act as the traffic controllers.

By matching a specialized retrieval token against the average Value projections of a frame, the model identifies semantic relevance much more accurately. As shown in the table below, using a target phrase in the Value space jumps Recall@1 from 65.7% to 78.0%.

Methodology: How ReToken Works

ReToken introduces a single learnable embedding ($X_r$) and a projection matrix ($W_r$).

1. Training: The VLM remains frozen. The ReToken is trained on multi-image QA datasets using a class-balanced binary cross-entropy loss. It learns to "summarize" the query and match the relevant visual features.
2. Inference (Two-Pass):
   • Pass 1 (Retrieve): The model encodes the video once. The ReToken is appended to the query and used to score all frames in the Value space at the final layer.
   • Pass 2 (Answer): The model re-runs the forward pass, but this time it only attends to the Top-K frames identified in Pass 1.

Experimental Results

ReToken's performance gains are most dramatic as the "haystack" gets larger. At a context size of 50 distractors, ReToken provides a 13.4% absolute accuracy boost over the vanilla Qwen3VL-8B.

More impressively, despite being trained only on static image collections, ReToken generalizes zero-shot to LVBench (long videos averaging 68 minutes), achieving an 8.0-point gain.

Critical Analysis & Conclusion

Strengths:

• Efficiency: Extremely lightweight (one token + one matrix).
• Plug-and-Play: Works with frozen, pretrained VLMs.
• Transferability: Strong zero-shot performance from images to video.

Limitations:

• Temporal Blindness: Since ReToken scores frames independently, it struggles with tasks requiring temporal reasoning (e.g., "what happened before the door opened").
• Summarization: It performs worse than uniform sampling on summarization tasks where evidence is dispersed across the entire video.

Final Takeaway: ReToken proves that we don't necessarily need massive architectural changes or fine-tuning to handle long visual contexts. By simply changing where we look in the Transformer (Value space) and how we ask (learnable tokens), we can significantly extend the reasoning capabilities of existing VLMs.