Bebop: Breaking the Entropy Bound to Supercharge RL Training

TL;DR

The Alibaba Qwen team has released Bebop, a system that solves the "acceptance rate collapse" in Reinforcement Learning (RL) rollouts. By proving that MTP performance is a slave to model entropy, they've replaced standard Cross-Entropy (CE) training with a specialized End-to-End TV Loss and Rejection Sampling. The result? A massive 1.8x end-to-end acceleration of RL training without the need for expensive online MTP updates.

The "Why": Why does MTP fail during RL?

Multi-Token Prediction (MTP) is a form of speculative decoding where a model predicts several future tokens at once. While this works beautifully in standard inference, it tends to break during RL.

The common wisdom was "Distribution Mismatch": as the LLM (target) updates its weights, the MTP (draft) heads get left behind. However, Bebop reveals a deeper, more fundamental villain: Policy Entropy.

As shown in Figure 1, there is a strict negative linear relationship between the model's entropy and the MTP acceptance length. In RL, we often increase entropy to encourage exploration. Standard MTP heads, trained with Cross-Entropy, are highly sensitive to this spreading of probability mass. When the model is "unsure" (high entropy), the MTP heads fail to guess the next tokens, and the speedup vanishes.

The "How": Rethinking the Objective (Total Variation)

Traditional MTP heads are trained to minimize KL Divergence. But speculative decoding performance is determined by the Total Variation (TV) distance, not KL.

Bebop introduces an End-to-End TV Loss. Unlike KL, which distributes optimization effort uniformly across the vocabulary (including low-probability "noise" tokens), the TV loss gradient is proportional to the token probability. This creates a Probability-Proportional Mismatch: the draft model focuses its limited capacity only on tokens that actually matter for acceptance.

The Bebop Recipe:

1. Rejection Sampling: Instead of greedy "Target-Only" sampling, Bebop uses Rejection Sampling to maximize distributional overlap.
2. e2e TV Loss: This loss directly optimizes the multi-step acceptance chain, weighting earlier steps more heavily.
3. Pre-RL Adaptation: The authors found that a well-trained TV-loss MTP head is so robust that it doesn't need to be updated during RL, saving massive amounts of VRAM.

Experimental Results: SOTA Efficiency

Across the Qwen 3.5, 3.6, and 3.7 families, Bebop consistently outperformed CE and KL baselines.

• In-Distribution Gains: Acceptance rates jumped by ~10% on reasoning and coding tasks.
• Agentic Breakthrough: On Agentic tasks, acceptance rates hit a staggering 95-98%, essentially making 4-token verification as fast as 1-token generation.
• Training Speed: In a full asynchronous RL pipeline, Bebop reduced per-step latency by 1.5x to 1.8x.

Critical Insight: The "Entropy-Invariant" Draft

The most profound takeaway from Bebop is that by using TV Loss, we can build MTP heads that are nearly entropy-invariant. In Figure 8 of the paper, the slope of the entropy-acceptance line drops by 95% (from -1.68 to -0.06). This means that even if the RL policy explores highly uncertain regions, the MTP heads continue to provide the same speedup.

Conclusion & Future Impact

Bebop demonstrates that the computational cost of "Rollout" in RL—long the bane of researchers—is solvable. By aligning the training objective (TV distance) with the inference mechanism (Rejection Sampling), the Qwen team has provided a blueprint for the next generation of efficient LLM training frameworks. As we move toward more complex agentic RL (like SWE-bench), Bebop's 1.8x speedup could save millions of GPU hours.