[ArXiv 2025] ReVal: Making LLM Reinforcement Learning Truly Off-Policy via Value-Based Learning

TL;DR

Reinforcement Learning for Large Language Models has long been enslaved by on-policy constraints—expensive trajectories are generated, used for one update, and thrown away. ReVal breaks this cycle by treating LLM logits as Q-values, enabling a value-based, off-policy framework with a replay buffer. The result? A 4.3x convergence speedup and significantly higher final accuracy on reasoning benchmarks like AIME and GPQA.

Problem & Motivation: The "Generation Bottleneck"

In the current RLVR (Reinforcement Learning with Verifiable Rewards) paradigm, the bottleneck isn't the weight update—it's the autoregressive generation. On-policy methods (like GRPO or PPO) require fresh data for every gradient step.

For complex reasoning tasks where tokens are many and rewards are sparse, this is incredibly wasteful. The authors identify a "Calibrated Initialization" problem in previous value-based attempts (like TBRM), where models would drift spuriously even when no reward was present. ReVal aims to fix the stability of value-based RL while finally enabling the reuse of historical data through a Replay Buffer.

Methodology: Logits as Q-Values

ReVal's breakthrough lies in its architectural simplicity. It avoids the overhead of a separate critic model by adopting the logit-as-Q parameterization.

1. The Unified Architecture

By interpreting the logits of the pretrained LLM as soft action-values (Q-values), the policy and value functions are fused. This preserves the "low-cost" advantage of actor-only methods while allowing for Bellman-style updates.

2. Calibrated Bellman Updates

Standard trajectory-level updates often suffer from instability. ReVal introduces a reward-shaping formulation that ensures that if the reward is zero, the optimal policy remains the reference policy. This "Calibrated Initialization" prevents the model from collapsing during the early stages of training.

3. Replay Buffer & Off-Policy Reuse

Unlike GRPO, which discards samples immediately, ReVal stores trajectories in a FIFO buffer. By performing multiple updates ($K$) per generation round, it extracts more signal from every generated token.

Experiments: Speed and Generalization

The authors tested ReVal against GRPO and TBRM on DeepSeek-R1-Distill-1.5B and Qwen2.5-Math-7B.

SOTA Performance

ReVal consistently outperformed on-policy baselines across mathematical reasoning tasks. Remarkably, the gap was largest on GPQA, a benchmark testing out-of-domain graduate-level reasoning, where ReVal beat GRPO by 4.5%.

Efficiency in "Extreme" Scenarios

In settings with limited rollouts ($N=1$), where data is scarcest, ReVal’s off-policy capability became its "superpower," reducing training time by 18% while maintaining higher accuracy than on-policy rivals.

Critical Insights: Why It Works

• KL Regularization Dynamics: The paper discovers that the KL term progressively weakens the gradient. ReVal counters this by periodically resetting the reference model, "reloading" the learning signal.
• Negative Samples Matter: Using normalized advantages (like GRPO) within the ReVal framework proved superior to binary 0/1 rewards, as it allows the model to explicitly suppress incorrect reasoning paths.

Conclusion & Future Outlook

ReVal proves that value-based RL is not just for Atari games; it is an essential tool for the next generation of LLM post-training. By shifting from "generate-and-discard" to "store-and-study," ReVal provides a blueprint for training long-horizon agents more efficiently.

Limitations: Currently, ReVal uses a simple FIFO buffer. Future work integrating Prioritized Experience Replay (PER) could likely push these efficiency gains even further by focusing the model on the most informative "hard" reasoning failures.