Tropical: Balancing the Tug-of-War Between Queuing and Interference in LLM Serving

TL;DR

In the high-stakes world of LLM serving, satisfying both TTFT (Time-to-First-Token) and TPOT (Time-per-Output-Token) simultaneously is notoriously difficult. Tropical breaks the stalemate between "non-disaggregated" (interfered but fast queue) and "disaggregated" (isolated but slow queue) architectures. By introducing SLO-aware multiplexing, it intelligently sneaks prefill tasks into decode workers using "latency slack," boosting request capacity by over 2x while maintaining strict service-level objectives.

The Architectural Dilemma: To Split or Not to Split?

LLM inference is a tale of two phases: Prefill (computation-heavy, generating the first token) and Decode (memory-bound, generating subsequent tokens).

1. Non-Disaggregated (e.g., vLLM): Prefills and decodes share the same GPU.
   • Problem: Large prefills "bully" decodes, causing massive spikes in TPOT (interference).
2. Disaggregated (e.g., DistServe, Mooncake): Prefills and decodes live on separate GPUs.
   • Problem: Prefill workers often become a bottleneck. Because decode workers cannot help, requests sit in long queues, destroying TTFT.

Tropical’s root cause analysis reveals a crucial insight: Disaggregated serving suffers from high queuing times, while non-disaggregated serving suffers from high interference.

Methodology: Mining Slack with SLO-Aware Multiplexing

Tropical doesn't just pick a side; it creates a dynamic bridge. The core innovation is the Multiplexing Toggle.

1. The Concept of "Slack"

In many serving scenarios, a decode worker might be operating faster than the required SLO. If the TPOT limit is 50ms but the worker is finishing in 30ms, there is 20ms of slack. Tropical monitors this slack in real-time.

2. The Multi-Path Dispatcher

Tropical routes requests through a dual-path system:

• Path 1 (Dedicated): Sends heavy prefill tasks to specialized prefill workers. This acts as an "interference inhibitor."
• Path 2 (Multiplexed): If a decode worker has enough slack, the toggle "piggybacks" a prefill request onto that worker's next iteration.

3. Dynamic Thresholding

The toggle is conservative. It predicts prefill execution time using offline profiling. It only triggers Path 2 if:

1. Predicted Interference < Total Slack.
2. HBM (Memory) usage is below a safe threshold.
3. TTFT SLO is at risk due to queuing.

Experimental Results: Reaching the Pareto Frontier

The authors evaluated Tropical using the Mooncake (InternLM-20B) trace on 8 NVIDIA A100 GPUs.

• SLO Attainment: Tropical supports 2.02x more users than baseline chunked-prefill methods before the SLO attainment drops below 90%.
• The Best of Both Worlds:
  • Vs. DistServe (Disaggregated): Tropical slashes P90 queuing time by 9x.
  • Vs. vLLM (Non-disaggregated): Tropical improves P90 TPOT by 2.8x.

The Cumulative Distribution Function (CDF) graphs clearly show that while other systems suffer from "tail latency" (extreme delays for a small % of requests), Tropical’s distribution is significantly tighter, indicating a more stable user experience.

Critical Analysis & Conclusion

The Takeaway

Tropical proves that the future of LLM serving isn't just about hardware isolation, but intelligent scheduling. By acknowledging that "interference" is acceptable as long as it doesn't violate the SLO, Tropical recovers wasted "slack" time that exists in static disaggregated systems.

Limitations & Future Work

While Tropical handles text-based dynamism brilliantly, future research could explore:

• Multi-Modal Variance: How does this multiplexing hold up when "prefills" (e.g., processing a large image) are significantly more heterogeneous?
• Hardware Heterogeneity: Adapting the multiplexing toggle for clusters with mixed GPU generations (e.g., A100s mixed with H100s).

Tropical sets a new standard for high-throughput, low-latency LLM serving by turning the rigid prefill-decode boundary into a flexible, SLO-aware interface.