Why fixed sparsity budgets feel uncontrollable—and what to do instead
Early block-sparse attention methods used a fixed top-k selection: every query keeps the same number of key blocks, regardless of how close the scores are. That feels uncontrollable because a query whose top-k cut is razor-thin can drop the one block that carries the answer. A 2026 study [2] showed this is a real failure mode: when the k-th and (k+1)-th blocks are nearly tied, the selector commits without spending extra budget, and the dropped block is unrecoverable. The fix is to make the budget adaptive—measure the score gap between the last kept and first rejected block, and for queries where that gap is small, double the kept set. This 'value-of-information' router lifted paired recall on a needle-in-a-haystack task from 0.51 to 0.63 at 32K context on a 14B model, and on a 7B model with 1M context it reached 1.00 at 32K and 0.81 at 128K, versus 0.09 for fixed top-k. The lesson: controllability comes from letting the model spend more compute when it's uncertain, not from a rigid cap.
Let the input decide the pattern, not a one-size-fits-all mask
Another lever is to adapt the sparse pattern itself to the input, rather than using a fixed structural mask. FlexPrefill [3] does this by measuring the Jensen-Shannon divergence between a query's attention distribution and a predefined pattern; if they diverge, it switches to a query-specific pattern. It also uses a cumulative-attention threshold to select indexes, ensuring the sum of attention scores meets a predefined threshold. This dynamic adjustment per head and per prompt is what makes the system feel controllable—you're not guessing which blocks matter; the model tells you. Similarly, VSPrefill [4] uses a lightweight indexer to predict importance scores for vertical columns and slash diagonals, then allocates sparsity budgets per layer with an adaptive cumulative-threshold strategy. It preserves 98.35% of full-attention accuracy while delivering a 4.95x average speedup at 128K context. Both approaches show that context-aware patterns give you a predictable accuracy-efficiency trade-off, which is the essence of control.
Don't throw away the long tail—compress it instead
A third design choice is to avoid the hard cutoff entirely by compressing unselected blocks into a compact state, rather than discarding them. SPLA [5] does this with a residual linear attention module: it computes the difference between global and selected linear attention, so unselected blocks are never explicitly accessed but their information is still captured. This closes the performance gap in continual pretraining and even surpasses dense attention on long-context benchmarks like RULER. The controllability here is that you don't have to worry about the 'only carrier of an answer hop' being lost—the long tail is still represented, just more cheaply. This is a different kind of control: you're not tuning a budget, you're changing the representation so that sparsity doesn't hurt as much.
About These Sources
This answer is built on 5 studies (2 peer-reviewed, 3 preprints) — published from 2024 to 2026, 5 from 2024 or later — selected as the most relevant from 6 studies that passed quality screening, drawn from 27 papers retrieved from a database of over 500 million.
Sources used in this answer
Block-Attention for Efficient Prefilling
Block-attention divides documents into blocks, computes KV states independently except for the final block, and after fine-tuning achieves TTFT of 45ms for 32K input, reducing TTFT by 98.7% and FLOPs by 99.8% versus full attention, while maintaining comparable performance.
Uncertainty-gated selection for block-sparse attention
An uncertainty-gated router that doubles the kept set when the top-k cut is uncertain lifts paired recall on RULER NIAH-multikey from 0.51 to 0.63 (mean) and 0.93 to 0.98 (Quest) at 32K on a 14B model, and reaches 1.00 on a 7B-1M model at 32K, with a budget-match ablation attributing roughly 2/3 of the lift to uniform-budget effect and 1/3 to selectivity.
FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
FlexPrefill dynamically adjusts sparse patterns and budgets per head using Jensen-Shannon divergence and cumulative-attention-based index selection, improving speed and accuracy over prior methods in long-sequence inference.
VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling
VSPrefill uses a lightweight VSIndexer to predict importance scores for vertical columns and slash diagonals, with an adaptive cumulative-threshold strategy, preserving 98.35% of full-attention accuracy while delivering a 4.95x average speedup at 128K context.
SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
SPLA uses a second-order Taylor expansion selection metric and a residual linear attention module to compress unselected blocks, surpassing dense attention on RULER while maintaining competitive general knowledge and reasoning.