What throughput and latency numbers should you actually compare?
A fair evaluation must measure both throughput (requests processed per second) and latency (time to first token and time between tokens) under the same workload. Paged attention can increase throughput by 2–4× compared to systems like FasterTransformer and Orca, but that gain is not free—it can introduce latency jitter if not carefully scheduled [4][3]. For instance, FlashInfer reports a 29–69% reduction in inter-token latency (the delay between generating each token) compared to compiler backends, which directly improves user-perceived speed [1]. However, chunked prefills—a technique that interleaves prefill and decode phases—can reduce generation stalls but may increase overall latency if not tuned properly [3]. So a fair test should report both throughput and latency percentiles (e.g., p50, p95) under identical request patterns.
How do you measure memory efficiency and scalability?
Memory efficiency is the core promise of paged attention, so a fair evaluation must quantify how much memory is saved and how that translates to larger batch sizes or longer sequences. Paged attention achieves near-zero waste in KV cache memory by eliminating fragmentation and enabling sharing across requests [4]. This allows more requests to fit in GPU memory, which is why throughput improves. But memory savings can be offset by quantization overhead—for example, quantizing the KV cache can reduce memory usage but may add computational cost, as seen in a vLLM-based study that improved throughput by up to 18% while maintaining accuracy [2]. A fair test should measure memory footprint per request, peak memory usage, and how batch size scales with memory constraints.
Why does the workload matter so much for a fair test?
Paged attention's benefits are highly workload-dependent, so a fair evaluation must use diverse, realistic workloads that vary sequence lengths, request arrival rates, and concurrency. The papers show that improvements are more pronounced with longer sequences, larger models, and more complex decoding algorithms [4]. For example, FlashInfer's 28–30% latency reduction for long-context inference highlights that gains are not uniform across all scenarios [1]. Additionally, integration with serving frameworks like vLLM and SGLang can affect results—FlashInfer's load-balanced scheduling is designed to handle dynamic user requests while maintaining compatibility with CUDAGraph, which requires static configuration [1]. A fair test should therefore include both kernel-level benchmarks and end-to-end serving benchmarks, as FlashInfer does, to capture the full picture.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2024 to 2025, 5 from 2024 or later — selected as the most relevant from 5 studies that passed quality screening, drawn from 48 papers retrieved from a database of over 500 million.
Sources used in this answer
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
FlashInfer, an attention engine integrated into SGLang, vLLM, and MLC-Engine, reports 29–69% inter-token latency reduction, 28–30% latency reduction for long-context inference, and 13–17% speedup for parallel generation compared to state-of-the-art serving solutions.
KV Cache Group Quantization Based on vLLM Inference Engine
A KV cache group quantization method based on vLLM improved inference throughput by up to 18% while maintaining low accuracy loss across LLaMA2-7B, LLaMA3-8B, and DeepSeek-R1-Distill-LLaMA3-8B models.
Efficient LLM Inference via Chunked Prefills
Chunked prefills, which split large prefill computations into smaller chunks and interleave them with decode operations, can improve serving capacity under strict latency constraints and reduce generation stalls, but require careful scheduling to avoid latency jitter.
Open-AI model Efficient Memory Reduce Management for the Large Language Models (LLMs) Serving with Paged Attention of sharing the KV Cashes
Paged attention achieves near-zero waste in KV cache memory and enables flexible sharing, improving throughput by 2–4× over FasterTransformer and Orca, with larger gains for longer sequences and larger models.
Fair, Practical, and Efficient Carbon Accounting for LLM Serving
A framework for carbon attribution in LLM serving uses Shapley values as ground truth and evaluates methods on scalability, fairness, sample efficiency, and incentivization, highlighting that simple leave-one-out methods fail to satisfy efficiency properties.