[FAIR 2026] Efficient RL Training for LLMs: Challenging the On-Policy Dogma

TL;DR

Reinforcement Learning (RL) has become the engine for LLM reasoning (think DeepSeek-R1), but it is prohibitively expensive because we discard rollouts immediately after one use. This paper from FAIR at Meta proves that Experience Replay (ER)—storing and reusing past trajectories—can slash compute costs by 40% without losing accuracy. In fact, "stale" data acts as a stabilizer, preventing the training crashes common in on-policy LLM RL.

The "Generate-then-Discard" Problem

In current SOTA RL pipelines (PPO, GRPO), inference workers generate fresh data, trainers run one gradient step, and the data is trashed. This is a massive bottleneck: inference often consumes 80%+ of total GPU hours.

The industry has avoided ER due to a fear of off-policiness: the idea that training on data generated by an older version of the model (stale data) introduces bias and noise that kills performance. This paper argues that this fear is mathematically and empirically overblown when weighed against the cost of generation.

Methodology: The Geometry of the Trade-off

The authors treat the RL pipeline design as an optimization problem. They decouple the Inference Workers (W) from the Trainers (T) using a replay buffer.

1. The Compute Ratio ($\gamma$)

The authors define $\gamma$ as the cost of a parameter update with a buffer versus without one. By increasing the ratio of trainers to workers, you replay samples more often, making every gradient step "cheaper" in terms of total GPU seconds.

2. Theoretical Framework

They model the training as a non-convex stochastic optimization problem, focusing on three variables:

• Staleness (${\bar{\sigma }}^2$): Noise introduced as data gets older.
• Coupling ($\rho$): How much replayed samples correlate with current parameters.
• Imbalance ($\mu$): The relative cost of rollout generation vs. training.

Figure: The asynchronous training setup where workers (W) and trainers (T) interact via a FIFO Replay Buffer.

Experiments: More than just Speed

The team tested their approach on Qwen2.5-7B and Qwen3-0.6B using the OpenR1-Math dataset.

Key Findings:

1. Efficiency Gains: A simple buffer saves 40% compute while reaching the same MATH accuracy as on-policy baselines.
2. Stability: On-policy training at high learning rates often "crashes" (accuracy drops to zero). Buffers act as a regularizer, smoothing out the objective function and preventing these collapses.
3. Diversity: Surprisingly, replaying older samples helps the model maintain output diversity, leading to better Pass@k scores (where $k>1$).

Figure: Accuracy on MATH vs. Compute. The orange (no-buffer) line is unstable and expensive; the buffer configuration (W=5, T=3) is faster and reaches a higher peak.

Replay Buffer Design: The Pareto Frontier

The authors performed a sweep across buffer sizes ($N$) and worker/trainer ratios. They found a clear Pareto frontier:

• Small Buffers: High diversity, low staleness, but low efficiency.
• Large Buffers: High efficiency and stability, but can eventually lag due to extreme staleness.

Pro-tip from the paper: Using "Positive-bias sampling" (keeping successful/correct trajectories in the buffer longer) further boosts performance, as correct reasoning steps are less sensitive to staleness than incorrect ones.

Critical Analysis & Conclusion

This work fundamentally shifts the RL post-training paradigm. It moves us away from the academic ideal of "pure on-policy learning" toward a more pragmatic, compute-centered engineering approach.

Limitations: The study primarily focuses on reasoning tasks (MATH, Coding). Whether extreme re-usage of samples holds up in more subjective "Alignment/Chat" tasks remains to be seen. However, for anyone building reasoning models (R1-style), the message is clear: Stop throwing away your data.

Takeaway: Experience Replay isn't just a trick for Atari games in 2015; it's a mandatory optimization for the LLM scaling era in 2026.