Why your throughput test may miss the network bottleneck
Most throughput benchmarks for LLM serving run on a single machine with fast NVLink or PCIe, so they assume KV cache transfer is limited by bandwidth. But in geo-distributed deployments—where prefill and decode run on different machines—the transfer of key-value (KV) cache is often limited by round-trip time (RTT), not bandwidth [1]. This happens because paged attention stores KV cache in non-contiguous memory blocks, and the current block-wise transmission mechanism sends them sequentially, which under high RTT behaves like the classic TCP small send window problem: you're waiting for acknowledgments instead of sending data, so effective throughput collapses even when the physical link is fast [1]. A throughput test that doesn't include network latency will completely miss this, making you think your system is faster than it actually is in production.
The best attention kernel isn't always the best serving schedule
Throughput tests often compare single-kernel implementations, like FlashInfer, and assume the fastest kernel gives the best serving performance. But a 2026 study shows that the best single-kernel implementation is not always the best serving schedule [3]. For long-context decode with low active batches, the kernel can under-utilize commodity GPUs, and mixed sequence lengths create a tension between exact-length launches and coarse padded batches [3]. The study's adaptive policy—which selects between FlashInfer, sequence splitting, and a workqueue schedule—improved synchronized wall throughput by 1.063–1.265x on B8 bimodal, uniform, and Zipf-like workloads, and by 1.399x on a B1 bucketed trace [3]. This means a throughput benchmark that only measures the kernel will miss the gains from smarter work assignment, which can be a decisive factor in real serving systems.
Throughput numbers hide how memory sharing and fragmentation behave
Paged attention's main selling point is near-zero waste in KV cache memory and flexible sharing of KV cache within and across requests, which can boost throughput by 2–4x compared to systems like FasterTransformer and Orca [2]. But that benefit is not uniform—it's more pronounced with longer sequences, larger models, and more complex decoding algorithms [2]. A simple throughput test with short sequences or simple decoding won't show you how paged attention handles the memory pressure of long-context requests. Similarly, a 2025 evaluation of vLLM (which uses PagedAttention) found that it significantly improves response time for chatbot inference, but the effect varies with the minimum number of tokens generated and the model used [4]. So a single throughput number can't tell you whether paged attention will actually help your specific workload—you need to test with your own sequence lengths and request patterns.
About These Sources
This answer is built on 4 studies (3 peer-reviewed, 1 preprint) — published from 2024 to 2026, 4 from 2024 or later — selected as the most relevant from 4 studies that passed quality screening, drawn from 34 papers retrieved from a database of over 500 million.
Sources used in this answer
RTT- or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving
In geo-distributed LLM serving, KV cache transfer is predominantly RTT-bound rather than bandwidth-bound, and the sequential transmission of non-contiguous paged-attention memory blocks causes effective throughput to collapse under high RTT, similar to the TCP small send window problem.
Open-AI model Efficient Memory Reduce Management for the Large Language Models (LLMs) Serving with Paged Attention of sharing the KV Cashes
PagedAttention achieves near-zero KV cache memory waste and flexible sharing, improving throughput by 2–4x over FasterTransformer and Orca, with larger gains for longer sequences, larger models, and more complex decoding algorithms.
PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
A page-aware decode scheduling study on an RTX 3060 found that an adaptive policy selecting between FlashInfer, sequence splitting, and workqueue scheduling improved synchronized wall throughput by 1.063–1.265x on B8 bimodal/uniform/Zipf-like workloads and 1.399x on a B1 bucketed trace, while avoiding regressions on a B4 boundary case.
Accelerating Chatbot Inference with vLLM: Evaluating the Efficiency of PagedAttention
An evaluation of vLLM with PagedAttention across Gemma, HuggingFaceTB, and Llama models found that it considerably improves response time for chatbot inference, with effects varying by model and minimum token generation settings.