[ArXiv 2026] IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

Executive Summary

TL;DR: IndexCache is a high-performance optimization for DeepSeek Sparse Attention (DSA) that slashes the computational cost of token selection by up to 75%. By identifying that consecutive Transformer layers attend to almost identical tokens, the authors replace redundant "lightning indexers" with a simple caching mechanism. The result is a massive 1.82x speedup in prefill and 1.48x in decoding for 200K-context tasks, with virtually zero impact on accuracy.

Background: In the landscape of efficient LLMs, sparse attention has moved from theory to production (e.g., DeepSeek-V3, GLM-5). However, even sparse models suffer from a hidden "quadratic tax": the indexer that decides which tokens to attend to. IndexCache is a "structural surgery" that solves this specific bottleneck.

The "Quadratic Tax" of Sparse Attention

Modern sparse attention models like DSA use a "Lightning Indexer"—a lightweight module that scores all preceding $L$ tokens to pick the top-$k$. While cheaper than full attention, it still scales at $O(L^2)$. In long-context scenarios (100K+ tokens), this indexer starts to eat the majority of the inference budget.

The authors observed a critical physical intuition: the top-k tokens selected at Layer 5 are 70–100% identical to those at Layer 6. Running a full indexer at every single layer is like checking a map at every step of a straight road—it's redundant.

Methodology: F-layers and S-layers

IndexCache redefines the inference loop by categorizing layers into two roles:

1. Full (F) Layers: These run the standard lightning indexer and "cache" the resulting top-k indices.
2. Shared (S) Layers: These skip the indexer entirely and simply "reuse" the indices from the previous F-layer.

Two Paths to Optimization

• Training-free (Greedy Search): Not all indexers are equal. Early layers are often "critical" for directing attention flow. The authors use a greedy algorithm to flip layers from F to S, keeping only those that minimize the Language Modeling (LM) loss on a small calibration set.
• Training-aware (Multi-layer Distillation): During training, an F-layer indexer is distilled not just against its own attention weights, but against the average attention distribution of all S-layers it serves. This forces the indexer to learn a "consensus" selection that works for the entire block.

Experimental Results: Breaking the 200K Barrier

The benchmarks on a 30B DSA model show that IndexCache delivers its highest value precisely where LLMs struggle most: long-context stability.

• Latency: At 200K context, prefill latency dropped from 19.5s to 10.7s (1.82x speedup).
• Throughput: Total decoding throughput increased by ~51% when the KV cache was fully utilized.
• Visual Evidence: The heatmap of index overlap confirms the "Block Structure" of LLMs, where certain layer clusters exhibit nearly 100% token selection similarity.

Deep Insight: Why Similarity Search Fails

An interesting "negative result" shared by the authors is that choosing which layers to skip based on Cosine Similarity of attention outputs (a common heuristic) actually performs poorly. Why? Because similarity is a local metric. It doesn't capture how small errors in token selection propagate and cascade through the network. The Greedy LM Loss Search is superior because it measures the global "end-to-end" impact of skipping an indexer, identifying "bottleneck layers" that are invisible to local similarity metrics.

Conclusion & Future Outlook

IndexCache proves that as we move toward "Agentic Workflows" requiring massive contexts, our focus must shift from just "core attention" to the "intelligence of selection." By exploiting cross-layer redundancy, we can maintain the accuracy of dense models with the speed of highly optimized sparse architectures.

Limitations: The training-free version requires a calibration set, and the optimal pattern may shift slightly if the model is used for radically different distributions (e.g., code vs. creative writing). However, the training-aware version largely mitigates this by making the model robust to uniform sharing.