DORA: Bridging the Gap Between System Throughput and RL Convergence for Reasoning LLMs

TL;DR

Reinforcement Learning (RL) has become the gold standard for "training" Large Language Models (LLMs) to reason. However, the rollout phase—where the model generates responses—is notoriously inefficient because of the "long-tail" problem: one long response can freeze the entire GPU cluster. DORA (Dynamic ORchestration for Asynchronous Rollout) solves this by allowing multiple policy versions to coexist in a streaming pipeline. It achieves up to 8.2x faster rollouts and 2.12x higher throughput without breaking the mathematical rules of RL.

The "Long-Tail" Saboteur in RL Training

In modern RLHF or GRPO training, the hardware spends 50-80% of its time just waiting for the model to finish speaking. Because reasoning models like DeepSeek-R1 or OpenAI's o1 generate massive Chain-of-Thought (CoT) trajectories, the distribution of response lengths is heavily skewed.

In a synchronous setup, if 63 GPUs finish their tasks but one GPU is still processing a "long-tail" reasoning chain, the entire system sits idle. This creates massive "bubbles" of wasted compute.

Figure: Traditional synchronous training creates massive idle "bubbles" while waiting for the longest response.

Methodology: The Philosophy of Constrained Optimization

DORA’s authors argue that we shouldn't just chase speed; we must respect the math. They identify three holy grails for RL stability:

1. C1: Intra-trajectory Consistency: Every token in a single response must come from the same model version.
2. C2: Data Integrity: You can’t just throw away long responses to save time—those are exactly the samples where the "reasoning" happens.
3. C3: Bounded Staleness: You can't train on data that is too old.

Multi-Version Streaming Training

Instead of waiting for everyone to finish, DORA streams completed trajectories to the trainer immediately. If a response is taking too long, it continues running on its original policy version while the rest of the cluster moves on to the next training iteration with an updated model.

Figure: The DORA timeline showing multi-version execution and non-blocking training.

System-Algorithm Co-Design: The KV-Cache "Free Lunch"

One of the most brilliant insights in DORA is how it handles resource migration. Usually, moving a running request from one GPU to another requires a "re-prefill" (re-calculating all previous tokens), which is incredibly expensive for long contexts.

However, because DORA enforces C1 (Intra-trajectory Consistency), the KV-Cache for a specific prompt is mathematically identical across any GPU hosting that version. DORA uses this to enable Zero Re-prefill Migration. It simply ships the KV-Cache over the network, allowing the model to "pick up where it left off" on a new device without wasting a single TFLOP.

Experimental Performance: Breaking Records

DORA was tested on Qwen2.5-32B and massive 560B MoE models. The results are striking:

• Rollout Stage: Up to 8.2x speedup compared to synchronous baseline.
• End-to-End Throughput: 2.12x improvement on 128 GPUs.
• Convergence: As shown below, DORA (k=3 staleness) tracks the reward curve of synchronous training almost perfectly, proving that "asynchronous" doesn't have to mean "unstable."

Figure: DORA maintains convergence parity with synchronous training while running significantly faster.

Critical Insight & Future Outlook

DORA proves that the "System vs. Algorithm" conflict is a false dichotomy. By strictly following RL constraints (like C1), we actually unlock system optimizations (like KV-Cache reuse) that aren't possible when those rules are relaxed.

Limitations: The "Staleness Bound (K)" still needs manual tuning. Future iterations might see an "Autopilot" for DORA that adjusts K dynamically based on the hardware's real-time pressure.

The Takeaway: For anyone building the next generation of reasoning models (o1-rivals), DORA offers the blueprint for a training system that doesn't blink when faced with the "long-tail" of intelligence.