What was broken before paged attention?
Before paged attention, LLM serving systems managed the key-value (KV) cache—the memory that stores the conversation context—as one big, contiguous block per request. That caused massive fragmentation: memory was wasted in gaps, and requests couldn't be batched together efficiently, limiting throughput. The original PagedAttention paper (2023) showed that by borrowing the operating system's idea of paging—splitting memory into fixed-size blocks—vLLM achieved near-zero waste in KV cache memory and improved throughput by 2–4× over state-of-the-art systems like FasterTransformer and Orca [1]. That's the foundation: paged attention turned a memory-management problem into a scheduling problem, and that shift is what enables the next wave of improvements.
How is paged attention evolving beyond the original design?
The original PagedAttention was designed for GPUs, but the same idea is now being adapted to other hardware and optimized further. A 2025 paper integrated PagedAttention with PyTorch's FlexAttention, showing that on an NVIDIA L4 GPU, inference latency grew only linearly (~2×) when sequence length increased from 128 to 2048 tokens, compared to exponential growth without caching [2]. That means long-context requests won't blow up response times as they used to. Another 2026 paper introduced Ragged Paged Attention (RPA) for Google's TPUs, achieving up to 86% memory bandwidth utilization in decode and 73% model FLOPs utilization in prefill on Llama 3 8B [3]. These numbers translate to faster, more cost-effective serving on cheaper hardware, which is exactly what API providers need to offer longer contexts without raising prices.
Why is the next big win in scheduling, not just kernels?
Even with a great attention kernel, the way you schedule work across requests can make or break throughput. A 2026 study, PersistentKV, found that the best single-kernel implementation isn't always the best serving schedule: for long-context decode with small batches, GPUs can be under-utilized, and mixing sequence lengths creates tension between exact-length launches and padded batches [4]. By using a page-aware scheduler that only executes non-empty tasks, they improved decode throughput by 1.04–1.08× on typical workloads and by 1.40× on a single-request long-context trace, while cutting launch overhead from 16 to 2 launches per step [4]. This shows that how you assign work is as important as the kernel itself—and that's where the next two years of optimization will focus.
What will users actually notice in LLM APIs?
For developers and end-users, the practical payoff is faster responses and lower costs. A 2025 study tested vLLM with Gemma, HuggingFaceTB, and Llama models and found that PagedAttention significantly reduced inference time by minimizing memory fragmentation and improving KV cache management [5]. Another 2025 paper traced the evolution from recommendation systems to LLMs, showing that PagedAttention is now a standard part of the infrastructure stack for serving LLaMA models [6]. Over the next two years, as these optimizations mature, expect APIs to handle much longer contexts (think entire books) without latency spikes, and to do so at lower cost per token—making advanced LLM features more accessible to startups and hobbyists alike.
About These Sources
This answer is built on 6 studies (3 peer-reviewed, 3 preprints) — published from 2023 to 2026, 5 from 2024 or later, collectively cited 776 times — selected as the most relevant from 6 studies that passed quality screening, drawn from 60 papers retrieved from a database of over 500 million.
Sources used in this answer
Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
The original PagedAttention paper (2023) introduced block-based KV cache management in vLLM, achieving near-zero memory waste and 2–4× throughput improvement over FasterTransformer and Orca, with larger gains for longer sequences and larger models.
Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
A 2025 study integrated PagedAttention with PyTorch's FlexAttention, showing on an NVIDIA L4 GPU that inference latency grew only linearly (~2x) from 128 to 2048 tokens with a global KV cache, versus exponential without caching.
PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
A 2026 paper presented Ragged Paged Attention (RPA) for TPUs, achieving up to 86% memory bandwidth utilization in decode and 73% model FLOPs utilization in prefill on Llama 3 8B, and integrated it as the primary TPU backend in vLLM and SGLang.
Efficient Memory Management for Large Language Model Serving with PagedAttention
A 2026 study, PersistentKV, showed that page-aware scheduling improves decode throughput by 1.04–1.08x on typical workloads and 1.40x on a single-request long-context trace, while reducing launch fan-out from 16 to 2 launches per step.
Scaling AI Infrastructure: From Recommendation Engines to LLM Deployment with Paged Attention
A 2025 evaluation of vLLM with Gemma, HuggingFaceTB, and Llama models found that PagedAttention significantly reduced inference time by minimizing memory fragmentation and improving KV cache management, making it suitable for real-time chatbots.
Accelerating Chatbot Inference with vLLM: Evaluating the Efficiency of PagedAttention
A 2025 article traced the evolution from recommendation systems to LLM deployment, presenting PagedAttention as implemented in vLLM as a key solution for memory management challenges, and provided a practical guide for serving LLaMA models.