Saguaro: Eliminating the Last Sequential Barrier in LLM Inference via Speculative Speculative Decoding

TL;DR

Even "Speculative Decoding" has been sequential—until now. Speculative Speculative Decoding (SSD), realized through the Saguaro algorithm, parallelizes the drafting and verification steps. By guessing the result of a verification before it finishes, Saguaro cuts out drafting overhead entirely on cache hits, delivering a 30% boost over traditional speculative decoding and 5x acceleration over standard autoregressive generation.

Background: The Hidden Sequential Gap

Traditional Speculative Decoding (SD) is the current gold standard for accelerating LLMs. It uses a small "draft" model to guess tokens and a large "target" model to verify them in one pass. However, there is a hidden flaw: The Draft model and the Target model work in a relay race. The draft cannot start speculating for round $T+1$ until the target has finished verifying round $T$. This sequential gap wastes precious milliseconds.

Saguaro asks a radical question: Can we eliminate this dependence?

Methodology: Anticipatory Speculation

Instead of waiting, Saguaro's draft model predicts the likely outcomes of the ongoing verification. Since verification can result in $0$ to $K$ tokens being accepted plus a "bonus token," the space of outcomes is large. Saguaro manages this through three primary technical pillars:

1. The Speculation Cache & Geometric Fan-Out

Saguaro builds a cache of pre-speculated sequences. To optimize this, it uses a Geometric Fan-Out strategy. Since shorter acceptance lengths are statistically more likely in low-accuracy draft models, the algorithm allocates more "guesses" (fan-out) to shorter prefixes.

Figure 1: High-level overview of the SSD framework where Speculator and Verifier operate on separate hardware.

2. Saguaro Sampling: Controlling the Residual

One of the hardest parts of SSD is predicting the "bonus token" generated by the target model when a token is rejected. Saguaro introduces a new sampling scheme that downweights the draft model's most likely tokens. This might seem counter-intuitive, but it forces the residual distribution (the distribution the target samples from upon rejection) to concentrate on a predictable set of tokens, significantly increasing the Cache Hit Rate.

3. Adaptive Fallback

At high batch sizes, cache misses become inevitable. Saguaro avoids stalling the entire batch by switching to a "Fast Backup Speculator" (low-latency or even random tokens) when the batch size exceeds a calculated threshold $b^{\ast }$.

Figure 2: Custom sparse attention mask used for multi-query decoding of all verification branches in parallel.

Experiments: Pushing the Pareto Frontier

The authors implemented Saguaro in a custom inference engine using PyTorch, PagedAttention, and NCCL.

• Performance: On Llama-3.1-70B, Saguaro reached 255.8 tok/s, compared to 161.8 tok/s for vanilla SD and 54.7 tok/s for autoregressive decoding.
• Scalability: While SD is typically seen as a latency-only play, SSD actually pushes the Pareto frontier for throughput as well, making it more compute-efficient per device at low and medium batch sizes.

Figure 3: SSD (Saguaro) consistently outperforms SD and AR across various datasets and improves the throughput-latency tradeoff.

Critical Insight: Why This Matters

The genius of Saguaro lies in its Hardware-Awareness. By placing the draft and target models on separate GPUs (e.g., Target on 4xH100, Draft on 1xH100), it exploits the inherent idle time of the draft model.

The transition from SD to SSD is analogous to the transition from sequential execution to out-of-order execution in modern CPUs. It treats LLM tokens not just as a linguistic sequence, but as a computational graph where branches can be pre-computed.

Conclusion

Saguaro proves that the "waiting period" in speculative decoding is not a fundamental law, but a bottleneck that can be engineered away. While it requires more FLOPs, the trade-off for significantly lower latency makes it a formidable contender for the next generation of real-time AI applications.

Limitations: The method relies on having separate hardware for the draft model and faces a "compute bound" limit at very large batch sizes where cache misses are frequent.