SAW-INT4: Bridging the Gap Between KV-Cache Quantization Theory and Real-World Serving
SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
SAW-INT4 introduces a system-aware 4-bit KV-cache quantization framework for LLMs that utilizes token-wise INT4 quantization combined with Block-Diagonal Hadamard Rotation (BDR). The method achieves near-lossless accuracy across various models (e.g., Qwen3, GLM-4) while maintaining the 4x memory reduction and high throughput of standard INT4 serving.
TL;DR
The memory footprint of the Key-Value (KV) cache has become the primary bottleneck for serving Large Language Models (LLMs), especially as context windows scale to millions of tokens. While many researchers propose complex compression algorithms, these often fall apart in production due to system-level constraints. SAW-INT4 solves this by using a "system-aware" approach: combining simple INT4 quantization with Block-Diagonal Hadamard Rotation (BDR). It recovers nearly all accuracy lost by naive quantization while maintaining maximum hardware throughput.
The Problem: The "System Gap" in Compression
Modern LLM serving engines (like vLLM or SGLang) rely on three pillars: PagedAttention, Continuous Batching, and Fused Kernels.
Current state-of-the-art compression methods often break these pillars:
- Token Eviction: Causes memory fragmentation because you can't free physical memory unless an entire "page" is evicted.
- Mixed-Precision: Breaks the uniform memory layout required for fast indexing.
- Vector Quantization (VQ): Requires codebook lookups that introduce irregular memory access, killing GPU performance.
The authors argue that for a method to be practical, it must be token-wise. It should operate on each token independently to stay compatible with paged layouts.
Methodology: The Power of Rotation
The core challenge of 4-bit quantization is outliers—specific channels with massive values that squash the quantization grid, leaving no resolution for the rest of the data.
1. Block-Diagonal Hadamard Rotation (BDR)
Instead of complex outlier management, the authors apply an orthogonal transform (Hadamard matrix) to the KV vectors. Since the transform is orthogonal, it preserves the 12-norm but "smears" the energy of outliers across all dimensions, making the data much easier to quantize.
To keep it efficient, they use a Block-Diagonal structure. Rather than one massive rotation, they split the head dimension into smaller blocks (e.g., 128).
2. Kernel Fusion
The researchers didn't just write a paper; they wrote a kernel. They fused the rotation and quantization directly into the Triton/CUDA decode kernels. By rotating the "Query" on the fly, they avoid the need to save intermediate rotated results to global memory, making the overhead virtually zero.
Figure 1: The SAW-INT4 pipeline showing the fusion of rotation into the existing serving stack.
Experiments: Near-Lossless and High-Speed
The results on the Qwen3 family are the most striking. Naive INT4 quantization usually results in a "collapsed" model (0% accuracy). With BDR-128, the accuracy returns to within 1-2% of the full BF16 precision.
Accuracy Highlights (Qwen3-4B):
| Method | Mean Accuracy | Accuracy Drop |
|---|---|---|
| BF16 (Baseline) | 75.64% | 0.00 |
| Naive INT4 | 0.00% | -75.64 |
| BDR-128 (SAW-INT4) | 73.78% | -1.86 |
Serving Efficiency
In real-world tests on H100 GPUs, SAW-INT4 matches the throughput of "Plain INT4" (which is inaccurate but fast) and significantly beats BF16 in system-wide Tokens Per Second (TPS).
Figure 2: System throughput (TPS) vs. Concurrency. Notice that SAW-INT4 (Green) tracks the speed of Plain INT4 (Red) while delivering high accuracy.
Critical Insights
- Complexity has Diminishing Returns: The authors tested Hessian-aware quantization and K-Means vector quantization. While mathematically "better," they provided only marginal gains over simple BDR once deployed, further proving that system compatibility is the "Gold Standard."
- KV-Cache is Model-Dependent: GLM-4.7 was surprisingly robust to naive INT4, while Qwen3 was extremely sensitive. BDR acts as a "safety net" that works for both.
- The Bottleneck is Bandwidth: Because LLM decoding is memory-bandwidth bound, adding a few extra FLOPs (arithmetic) for rotation is "free" as long as you don't add extra Memory Accesses (I/O).
Conclusion
SAW-INT4 provides a blueprint for the future of LLM deployment. It proves that the most "elegant" mathematical solution isn't always the best "engineering" solution. By respecting the constraints of paged memory and fused kernels, SAW-INT4 enables 4x KV-cache compression with zero measurable overhead.
Takeaway: If you want to scale LLM context lengths in production, look at token-wise rotation—it's the simplest path to SOTA efficiency.