Prefill-as-a-Service: Breaking the Datacenter Boundary for Next-Gen LLM Serving
Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter
Prefill-as-a-Service (PrfaaS) is a novel cross-datacenter LLM serving architecture that decouples compute-intensive prefill from memory-intensive decode across loosely coupled clusters. By leveraging hybrid-attention models (e.g., Kimi Linear) to reduce KVCache size, it enables transferring KV states over commodity Ethernet, achieving SOTA throughput in heterogeneous environments.
TL;DR
The "bandwidth wall" has long restricted Large Language Model (LLM) serving to single, tightly-coupled RDMA clusters. Prefill-as-a-Service (PrfaaS) changes the game. By exploiting the reduced KVCache footprint of next-generation hybrid-attention models, PrfaaS allows prefill tasks to be offloaded to remote, compute-dense clusters over standard commodity Ethernet. This architecture boosts throughput by 54% and slashes latency by 64%, enabling a truly global, heterogeneous serving infrastructure.
The Motivation: Escaping the RDMA Island
In the current LLM landscape, Prefill-Decode (PD) Disaggregation is the standard. Prefill (processing the prompt) needs raw FLOPs, while Decode (generating tokens) needs memory bandwidth. However, these two phases are "married" to the same rack or datacenter because transferring the resulting KVCache is incredibly heavy.
For a standard dense model (like MiniMax-M2.5), a single 32K token request can generate KVCache at 60 Gbps. To handle this, you need a high-speed RDMA fabric. This creates two major problems:
- Hardware Inflexibility: You can't use specialized prefill chips (like NVIDIA Rubin CPX) if they aren't in the same physical cluster as your decode chips (like LPUs).
- Stranded Capacity: You can't easily "borrow" idle compute from another region because the network won't keep up.
(a) Traditional coupled single-cluster inference vs. (b) The new PrfaaS paradigm.
Methodology: The Hybrid Advantage
The authors realize that new model architectures—Hybrid Attention—provide a "loophole" in the bandwidth wall. Models like Kimi Linear or Qwen 3.5 interleave standard attention with linear or sliding-window attention. This reduces the KVCache size by 10x to 36x.
1. Selective Offloading
PrfaaS doesn't send everything to the remote cluster. Short requests are memory-bound; sending them over a network is a waste. PrfaaS uses Length-Based Threshold Routing:
- Short Requests: Processed locally in the PD cluster.
- Long Requests: Sent to the high-compute PrfaaS cluster.
2. The Hybrid Prefix Cache Pool
Managing KVCache becomes complex when different layers have different storage needs. PrfaaS introduces a unified block pool that treats linear states (fixed size) and full-attention KVCache (dynamic size) separately, allowing for clever cross-cluster reuse.
Figure: The Hybrid Prefix Cache Pool categorizes blocks for intra-cluster reuse or cross-cluster transfer.
3. Dual-Timescale Scheduling
The system handles the "burstiness" of internet traffic through two layers:
- Short-term: Monitors egress utilization to adjust routing thresholds instantly, preventing network congestion.
- Long-term: Periodically re-profiles traffic to balance the ratio of prefill vs. decode instances.
Experimental Results: Efficiency Reimagined
Testing on an internal 1T-parameter hybrid model, the researchers found that their architecture transforms commodity Ethernet into a viable backbone for KV transfer.
| Metric | PrfaaS-PD | Homogeneous PD | Improvement |
|---|---|---|---|
| System Throughput | 3.24 req/s | 2.11 req/s | +54% |
| P90 TTFT | 3.51 s | 9.73 s | -64% |
| KV Egress Load | 13 Gbps | N/A | Feasible on Ethernet |
Optimization search: The system finds the perfect threshold (t) to balance compute capacity and network bandwidth.
Critical Analysis & Conclusion
Takeaway: This paper successfully argues that the future of LLM serving is not just about faster chips, but about architectural co-design. By shrinking the KVCache at the model level, we unlock massive flexibility at the systems level.
Limitations:
- The efficiency depends heavily on the model architecture. If you are stuck with "pure" Transformer models, the bandwidth requirements remain prohibitive.
- Latency (Speed of Light) between datacenters still poses a lower bound on TTFT, meaning clusters shouldn't be too geographically distant (e.g., cross-continent might still be tricky).
Looking Forward: As context windows expand to millions of tokens, PrfaaS provides the blueprint for "Cloud-Native Inference," where compute capacity can be dynamically gathered from across the globe to handle the world's most demanding prompts.