Why throughput tests miss the time-to-first-token problem
Throughput tests typically measure how many tokens the system generates per second once generation starts—but for long-context requests, the real user pain is the wait before the first token arrives. That wait is called time-to-first-token (TTFT), and it's dominated by the prefill phase, where the model processes the entire prompt. Block-sparse attention can slash this: one method cut TTFT by 98.7% and FLOPs by 99.8% for a 32K-token input, bringing first-token latency down to 45 milliseconds [1]. A serving system that only reports tokens-per-second would never reveal that a user waited 45 ms instead of several seconds—because that latency happens before generation even begins.
Another study shows the same gap from a different angle: a sparse-attention serving system reduced mean TTFT by up to 9.26x compared to state-of-the-art systems, while boosting generation throughput by 3.14x [3]. If you only looked at the throughput number, you'd think the system was only modestly better—but the TTFT improvement is the headline for interactive long-context use. So throughput tests can make a system look fine when it's actually terrible at the thing users notice first.
Throughput tests hide the memory pressure that limits batch sizes
Even when sparse attention skips computing on most tokens, the key-value (KV) cache for those unselected tokens still has to stay in high-bandwidth memory (HBM) for fast decoding later. That means sparse attention doesn't automatically free up memory—it just shifts the bottleneck from compute to memory capacity. One study found that this HBM capacity constraint limits how many requests you can batch in parallel, which directly caps throughput [3]. A throughput test that doesn't account for this will overestimate how many concurrent long-context requests the system can actually handle.
The same paper shows that offloading underutilized KV caches to DRAM can free HBM and allow larger batch sizes, but that requires careful management—fragmented cache access, HBM contention, and hybrid batching all become issues [3]. So a naive throughput benchmark might show a sparse-attention system achieving high tokens-per-second on a single request, but in a real multi-request serving scenario, memory pressure forces smaller batches and the throughput collapses. That's exactly the kind of failure a simple throughput test won't catch.
Throughput tests ignore the accuracy cost of aggressive sparsity
Block-sparse attention speeds things up by skipping tokens it deems unimportant—but if the selection is wrong, you lose accuracy. One study found that a vertical-slash sparse method preserved 98.35% of full-attention accuracy on long-context benchmarks while delivering a 4.95x speedup at 128K context [4]. That sounds great, but it also means a 1.65% accuracy drop—which a throughput test would never show. If you're serving a model for a task where that 1.65% matters (like legal or medical reasoning), the speedup is worthless.
Another method, SPLA, explicitly addresses the accuracy problem by compressing unselected blocks into a recurrent state instead of discarding them, and it actually surpassed dense attention on the RULER benchmark [5]. But that's the exception—most sparse methods trade accuracy for speed, and the trade-off is invisible in throughput numbers. A throughput test that reports tokens-per-second without measuring task performance gives a dangerously incomplete picture of whether the system is actually usable.
About These Sources
This answer is built on 5 studies (2 peer-reviewed, 3 preprints) — published from 2024 to 2026, 5 from 2024 or later — selected as the most relevant from 5 studies that passed quality screening, drawn from 24 papers retrieved from a database of over 500 million.
Sources used in this answer
Block-Attention for Efficient Prefilling
Block-attention reduced TTFT by 98.7% and FLOPs by 99.8% for a 32K-token input, achieving 45 ms first-token latency, while maintaining performance comparable to full attention across 11 benchmarks.
FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving
FlashPrefill V2 achieved up to 47.26x and 27.19x speedups over FlashAttention-2 at 128K context under FP8 and BF16, respectively, and 30.49x over a dense FA3/4-aligned baseline in FP8, while introducing a mean correction term to suppress approximation error.
SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
SparseServe reduced mean TTFT by up to 9.26x and increased token generation throughput by up to 3.14x versus state-of-the-art systems, by managing HBM-DRAM KV cache offloading to enable larger batch sizes.
VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling
VSPrefill preserved 98.35% of full-attention accuracy on LongBench and RULER while delivering a 4.95x average speedup at 128K context, using a lightweight indexer and adaptive sparsity budgets.
SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
SPLA, using second-order Taylor expansion for block selection and residual linear attention for unselected blocks, surpassed dense attention on RULER while maintaining competitive general knowledge and reasoning.