Can vision token compression work with linear attention in VLMs?

Existing compressors fail on hybrid linear-attention VLMs

Vision tokens dominate the input sequence of a vision-language model, so cutting them is a direct way to reduce computation. The paper's starting problem is that the established tools for doing this were built for softmax attention. Methods that rank tokens by attention scores, or that merge tokens by similarity, assume properties of softmax attention that a hybrid architecture does not preserve.

The authors report systematic experiments in which both attention-based and similarity-based compression fail to surpass even a random baseline once moved to hybrid architectures, and produce suboptimal results. Their explanation is an information-theoretic ceiling imposed by the bounded recurrent state of linear attention. That bound disrupts the shallow-layer attention concentration that attention-based methods exploit, and the token distinguishability that similarity-based methods need.

The same analysis turns into the paper's opening: linear attention's selective retention itself encodes a signal about which tokens matter. V-CoLA is built on that signal rather than on borrowed softmax statistics.

Visual information fades with depth and sequence length

To test whether the recurrent state really tracks importance, the authors measure a retention rate: how faithfully information written into the final state can be reconstructed, across vision token lengths and model depths, on Qwen3.5-9B. They also record the Pearson correlation of retention-rate rankings between adjacent layers.

Three observations follow. Retention is relatively high in shallow layers and declines with depth, dropping below 20% in deeper layers, which suggests visual information is heavily used in shallow reasoning and increasingly marginalized later. Retention also falls as sequence length grows, consistent with the state being under pressure given its limited size. And the rank correlation between adjacent layers stays high in shallow layers and declines in deeper ones, implying that during visual reasoning different layers share consistent preferences about which tokens are critical.

This is why the method reads importance from shallow layers: that is where the state's selectivity is most informative and most consistent across depth.

Importance combines reconstruction error with uniqueness

V-CoLA scores each vision token with two terms. The first, an importance term, is one minus a normalized cosine distance between the token's value and what the final state can reconstruct for its key. A low reconstruction error means the token's information survived in the state, so the token mattered.

Filtering on that term alone biases selection toward salient regions and yields many redundant tokens, and tokens whose contribution is attenuated by accumulated gated decay in the final state can be missed. The second term, uniqueness, addresses this by probing how much the state's readouts change when a token is absorbed. A set of pseudo queries drawn from nearby tokens reads value vectors from the state before and after the update, and the average normalized distance becomes the uniqueness score. Redundant tokens that follow similar ones move the state less.

The final score is a weighted sum of the two terms, with a weight controlling the uniqueness share. Because both terms are normalized distances over value-space readouts, the combined score stays in the same range and interpretation.

Merging beats pruning and stays compatible with chunked parallelism

Compression does not simply keep the top-scoring tokens. The paper cites recent analysis indicating that token aggregation is more robust than token pruning, and adopts an adaptive chunk-wise merging strategy instead of top-k retention.

The engineering constraint is that linear attention runs in chunks to parallelize its recurrence. The authors integrate the importance computation into that chunk-wise parallelism and extend Gated DeltaNet with a bypass input and batched queries so extra queries can be injected during output computation, using shifted keys as pseudo queries. They report this incurs minimal additional runtime overhead. All components are training-free, so no retraining of the base model is required.

Accuracy holds at half the tokens and degrades gracefully below

Across multiple benchmarks, V-CoLA reaches 99.5% of the original model's performance with only 50.0% of vision tokens, and over 88.0% with as few as 12.5%. These are retention ratios against the uncompressed model, not absolute scores, and the headline figures are aggregated across the benchmark suite rather than tied to one dataset.

The long-form evidence is more granular. On MMDU, a multi-turn multi-image dialogue benchmark with up to 20 images, 27 turns and 18K multimodal tokens per sample, V-CoLA stays within 1.4 points of the full model on all six scored dimensions even at a 12.5% token budget. Overall score moves from 66.6 at full tokens to 67.0 at 50.0%, 66.0 at 25.0% and 65.4 at 12.5%.

Fine-grained recognition shows where the budget matters. On all 191 V*-Bench samples, exiting at layer 24 stays within 1.05 points of the full model, while exiting at layer 12 loses 2.62 points. Under a matched 50% budget, early exiting reallocates deep-layer tokens to shallow layers and improves overall accuracy by 2.09 points.

Speedup grows faster than the token cut in softmax-heavy regimes

The reported prefill speedup ranges from 1.86x to 6.15x. The paper derives why the gain is not simply the inverse of the retention ratio: in Qwen3.5's 3:1 hybrid layout, three of every four layers use linear attention whose cost is linear in sequence length, while the remaining softmax layer is quadratic.

The closed-form analysis interpolates between 1/r and 1/r-squared depending on how much of the per-layer cost is quadratic, and is scaled by the fraction of layers that sit after the compression point. Empirically, at a retention ratio of 0.5 with 8K vision tokens the measured speedup is 1.91x against a theoretical 2.0x, and at 0.125 it is 6.15x against a theoretical 7.0x.

The gap between measured and theoretical values is itself informative: the theory bounds what token reduction can buy, and implementation overheads consume part of it.

The result is scoped to hybrid architectures with periodic softmax layers

The evaluation centers on hybrid VLMs that interleave linear-attention and softmax-attention layers, such as Qwen3.5 and InfiniteVL. The authors state that purely Mamba-based or fully linear-attention VLMs are not covered, and that extending the method there may require revisiting how the uniqueness-aware criterion interacts with the absence of periodic softmax layers.

The practical priority follows from the comparison inside the paper rather than from the headline number. If the deployment target is a hybrid linear-attention VLM, the relevant choice is not whether to compress but which signal to rank tokens by: attention scores and similarity both fall to or below a random baseline in this regime, while state-derived importance holds 99.5% of performance at half the tokens. If the target is a fully linear or Mamba-only model, that evidence does not yet extend, and the layer at which compression is applied remains a separate lever, since exiting at layer 24 preserved fine-grained accuracy far better than exiting at layer 12.

Sources used in this answer