Why memory fragmentation is the first thing that breaks in production
In production, the KV cache (the stored key-value pairs from previous tokens) grows and shrinks per request, and if managed poorly, it wastes memory through fragmentation and duplication, limiting how many requests you can batch. The original PagedAttention paper (2023) showed that existing systems waste significant memory this way, and by borrowing virtual-memory paging techniques, vLLM achieves near-zero waste and boosts throughput by 2–4× over state-of-the-art systems like FasterTransformer and Orca [5]. That throughput gain is the whole point: more requests batched means lower cost per request, which is why paged attention became the standard.
But the fix isn't free. A 2024 study introducing vTensor found that even with paged attention, memory fragmentation persists in practice—vTensor's virtual-memory approach freed about 71% (57GB) of GPU memory on an A100 compared to vLLM, enabling more memory-intensive workloads [2]. That's a huge difference: if you're running large models, that extra memory could mean the difference between fitting a batch or hitting out-of-memory errors. So the first production failure mode is not just fragmentation itself but the fact that paged attention doesn't eliminate it entirely—it reduces it, but you still need to monitor for residual waste.
What works on GPUs may fail on TPUs or other accelerators
Paged attention was designed for GPUs, and moving to other hardware like TPUs introduces new failure modes because the memory model and kernel execution differ. A 2026 paper on TPU inference found that existing paged attention kernels are GPU-centric and don't map well to TPUs, so they built a specialized kernel (Ragged Paged Attention) that achieves up to 86% memory bandwidth utilization in decode and 73% model FLOPs utilization in prefill on TPU7x [1]. Without such specialization, you'd likely see severe underutilization and poor performance on TPUs.
This is a critical production consideration: if you're deploying on TPUs (common for cost efficiency), you can't just assume vLLM's paged attention will work—you need a kernel tuned for that hardware. The same paper integrated their kernel into vLLM and SGLang as the primary TPU backend, showing that production-grade paged attention on TPUs requires significant engineering effort [1]. So the failure mode is not just about memory but about hardware-software co-design; ignoring this can lead to catastrophic performance drops.
About These Sources
This answer is built on 5 studies (1 peer-reviewed, 4 preprints) — published from 2023 to 2026, 4 from 2024 or later, collectively cited 776 times — selected as the most relevant from 9 studies that passed quality screening, drawn from 54 papers retrieved from a database of over 500 million.
Sources used in this answer
Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
Ragged Paged Attention, a TPU-specific kernel, achieves up to 86% memory bandwidth utilization in decode and 73% model FLOPs utilization in prefill on TPU7x, demonstrating that paged attention requires hardware-specific optimization to work efficiently on TPUs.
vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving
vTensor, using GPU virtual memory management, achieves up to 3.27× speedup over vLLM's paged attention kernel and frees ~71% (57GB) of memory on an A100, showing that paged attention still has kernel overhead and residual fragmentation.
ML and systems co-design for resource-efficient LLM inference serving
This thesis explores model-level optimizations (prompt compression, quantization) and system components built on vLLM, but notes that full integration into a complete serving system remains ongoing work, so it doesn't directly address production failure modes.
Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
Integrating PagedAttention with FlexAttention in IBM's FMS reduces latency growth to ~2× from 128 to 2048 tokens, but paged attention's power-of-two allocations cause incremental memory usage only at sequences over 2048 tokens.
Efficient Memory Management for Large Language Model Serving with PagedAttention
The original PagedAttention paper (2023) shows vLLM achieves near-zero KV cache waste and 2–4× throughput improvement over FasterTransformer and Orca, with larger gains for longer sequences and larger models.