What's been overturned: fixed budgets and block sizes are not optimal
The old assumption was that you could pick a single tree size or block size and it would work well for all inputs. That's now known to be wrong. [1] shows that acceptance length always increases with tree budget, so there's no principled way to choose a fixed budget—you'd always want the biggest tree, but that ignores verification cost. [3] shows the optimal block size varies per sample and clusters around the training size, so a one-size-fits-all approach leaves speed on the table. Together, these papers overturn the idea that a static configuration is good enough.
What needs solving: cost-aware tree construction, adaptive block size, and handling randomness
First, you need to build the draft tree with an eye on actual throughput, not just acceptance length. [1] introduces a method that jointly selects tree structure and node budget, modeling draft and verification latencies, and shows the throughput function is unimodal, allowing an efficient greedy stopping rule. This means you can adapt the budget each round without offline search, matching or beating an oracle that knows the best budget in advance.
Second, you need to adapt the block size to the input. [3] shows that predicting the optimal block size from the prefilling representation—the part of the input processed before generation—can yield a 4.20x speedup on Qwen3-4B, with an acceptance length of 5.92 tokens. This is a plug-and-play improvement that adds minimal overhead.
Third, you need to handle the target model's randomness. [4] shows that block diffusion drafters that assume conditional independence degrade as the target sampling distribution becomes more stochastic (higher entropy). They propose a dependent block drafter with an acceptance-oriented training objective that improves accepted length, especially in high-entropy regimes. This is crucial because real deployments often use sampling, not greedy decoding.
The new picture: dynamic, cost-aware, and stochastic-aware drafting
The emerging consensus is that block diffusion speculative decoding must be dynamic and cost-aware. [5] combines an acceptance surrogate, an online latency estimator, and adaptive best-first expansion to grow the tree until marginal gains no longer justify verification costs. It achieves up to a 6.61x speedup over autoregressive decoding and outperforms state-of-the-art baselines by 39%, without per-setting tuning. This shows that training-free, budget-aware methods can be both effective and practical.
Another angle is using multiple drafters to recover from early mismatches. [2] proposes a dual-diffusion framework that uses confidence scores to identify likely rejection boundaries and re-anchor alternative continuations, improving acceptance rates over single-sequence drafting. This addresses the problem that committing to a single draft sequence wastes all subsequent tokens after the first mismatch.
The key takeaway is that the field is moving from static, greedy, cost-blind drafting to adaptive, cost-aware, and stochastic-aware methods. The papers here agree that dynamic decisions—whether about tree budget, block size, or handling randomness—are essential for real-world speedups. The strongest evidence comes from [1] and [3], which show concrete gains from adaptation, while [4] and [5] highlight the importance of handling randomness and hardware costs.
About These Sources
This answer is built on 5 peer-reviewed studies — published in 2026, 5 from 2024 or later — selected as the most relevant from 5 studies that passed quality screening, drawn from 30 papers retrieved from a database of over 500 million.
Sources used in this answer
Cost-Aware Diffusion Draft Trees for Speculative Decoding
CaDDTree optimizes token throughput by jointly selecting tree structure and node budget, proving the throughput function is unimodal under convex verification cost, and matches or surpasses an oracle budget selection on Qwen3-4B/8B across eight benchmarks.
D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
D^2SD uses two diffusion drafters—one to generate a block with confidence scores and another to re-anchor at likely rejection boundaries—improving acceptance rates over single-sequence drafting.
BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decoding
BlockPilot predicts the optimal block size per input from the prefilling representation, achieving a 4.20x speedup and 5.92 acceptance length on Qwen3-4B at temperature 1, with minimal overhead.
DBLAST: Dependent Block Drafting for Stochastic Speculative Decoding
DBLast shows that block diffusion drafters degrade as target sampling entropy increases, and proposes a dependent block drafter with an acceptance-oriented objective that improves accepted length, especially in high-entropy regimes.
Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting
BASTION is a training-free, budget-aware framework that dynamically grows a tree until marginal gains no longer justify verification costs, achieving up to a 6.61x speedup and outperforming state-of-the-art baselines by 39% across diverse benchmarks and GPU architectures.