RRL: Solving the "Early Suppression" Paradox in LLM Reasoning

TL;DR

Reinforcement Learning (RL) for LLMs is often a race against time: as the model gets better at following a specific policy (Exploitation), its ability to try new, diverse ideas (Exploration) collapses. Retrospective Replay-based Reinforcement Learning (RRL) fixes this by allowing models to go back in time. By replaying "promising" thoughts the model had when it was younger and weaker, RRL gives the mature model a second chance to turn those early sparks of intuition into correct, reward-winning solutions.

Background Positioning: This work is a strategic refinement of the RL post-training pipeline (like PPO), specifically targeting the "forgetting" issue in complex reasoning domains like Math and Code.

The Problem: Why LLMs "Lose Their Curiosity"

In tasks like code generation, a single wrong character at the end of 500 lines makes the reward zero. During early training, an LLM might have a brilliant idea for a function (a "promising state") but fail to close the brackets correctly. The Policy Gradient looks at the result (Failure), sees the path, and says: "Don't ever go down this road again."

As training progresses:

1. Policy Narrowing: The model becomes more confident but less diverse (KL-divergence shrinks).
2. Exploration Decay: By the time the model is "smart" enough to write the code perfectly, it has already been trained to avoid the very concepts needed to solve the hard problems.

Figure: How Policy Gradient suppresses early promising ideas before the model can master them.

Methodology: The RRL Framework

RRL transforms the training loop into a "Time Machine" using three core components:

1. Identifying "Promising States"

The system doesn't just save everything. It uses the Value Model to score intermediate segments of text. If a partial solution has a high predicted reward, it’s saved to a buffer. It captures these from two sources:

• Policy-Generated Solutions (PGS): "I almost had it!"
• Canonical Solutions (CS): "This is how the pros start."

2. The Retrospective Replay Mechanism

Instead of starting every training sample from the prompt $q$, the model has a probability $p$ (the Replay Coefficient) to start directly from a saved state $s$ in the buffer. This bypasses the early stages of exploration where the model might have previously tripped up.

3. Masked Optimization

When replaying from an intermediate state, RRL masks the tokens that make up the "state" itself. This prevents the model from over-optimizing (and potentially ruining) the starting point, focusing the gradient entirely on the new exploration path.

Figure: The RRL training loop showing the buffer interaction and PPL variance.

Experiments and Results

RRL was tested against standard PPO and industry-standard models like DeepSeek-Coder.

Significant Gains in Coding

On the APPS+ dataset, RRL outperformed vanilla PPO by 3.5%. Notably, the gains were highest in the "Interview" and "Competition" categories—the hardest problems where exploration is most likely to fail.

Better Alignment (RLHF)

In general dialogue tests (HH-RLHF), GPT-4 was used as a judge. RRL achieved a 74.2% win rate against SFT models, proving that better exploration isn't just for math—it helps the model find safer and more helpful ways to communicate.

Critical Insights & Future Outlook

The beauty of RRL lies in its simplicity: it acknowledges that learning is non-linear.

• Ablation Success: The paper shows that using both user-generated and canonical states is better than either alone. Using only canonical states turns RL into "Imitation Learning," while using only generated states risks replaying errors.
• Limitations: Currently, the Value Model's inaccuracy in early training can pollute the buffer. Future work will likely involve "Metacognition"—teaching the model to doubt its own value estimates before saving a state.

Conclusion: RRL proves that the secret to better reasoning isn't just more data, but a better way to remember the "what ifs" from early in the learning process.