[ICLR 2025] LongRLVR: Solving the Vanishing Gradient in Long-Context Reinforcement Learning

Executive Summary

TL;DR: Reasoning over 100k+ tokens is hard not just because of memory, but because of optimization. Standard Reinforcement Learning (RL) fails here because if the model misses one key sentence in a sea of text, it gets a zero reward, leading to a "vanishing gradient" where the model never learns to find information. LongRLVR fixes this by introducing a verifiable context reward, teaching the model exactly which "chunks" of text are useful before it even tries to answer.

Positioning: This work moves beyond simple context window extension (like RoPE scaling) into the territory of Long-Context Alignment, proving that we can "train" models to be better explorers of their own internal context.

The Problem: The Sparse Reward Trap

In standard RLVR (used by models like DeepSeek-R1), the model is rewarded if the final answer is correct. This works for math because the reasoning steps are internal. However, for long-context tasks, the model must first perform Contextual Grounding.

The authors identify a fatal flaw: The Vanishing Grounding Gradient.

• If a question requires 3 specific facts (chunks) to answer, and the model only finds 2, it gets 0 reward.
• Because it gets 0 reward, the gradient is 0.
• The model never learns that those 2 chunks were actually helpful.

Figure 1: Notice how Naive RLVR (purple) hits a ceiling in both recall and accuracy, while LongRLVR (red) continues to improve.

Methodology: Rewarding the "Search"

LongRLVR factorizes the LLM's task into two heads (logical, not physical architecture):

1. Grounding Head: Identifies chunks $Z$ from context $X$.
2. Answer Head: Generates answer $y$ based on $Z$.

The Secret Sauce: Modulated F-Score

Instead of just rewarding the answer, the authors use a verifiable reward for the grounding step. They use an $F_{\beta }$-score (where $\beta =2$ to prioritize recall): $r_{\mathrm{ctx}}=\eta \cdot F_{\beta }+(1-\eta )\cdot r_{\mathrm{ans}}\cdot F_{\beta }$

This formula is clever: it gives a small "pity reward" for finding the right chunks even if the answer is wrong ($\eta \cdot F_{\beta }$), but unlocks a massive "synergy reward" if the model finds the right chunks and gets the answer right.

Figure 2: The model output format: <useful chunks> identifiers are generated before the <answer>.

Experiments and Results

The authors tested LongRLVR on 128k context benchmarks. The results are striking:

• Efficiency: A 14B Qwen model trained with LongRLVR outperformed the 72B version of the same model and even rivaled the specialized Qwen3-14B (Thinking) model.
• Robustness: The model's performance remained stable whether the document was split into 16 chunks or 128 chunks, proving it learned semantic relevance rather than just pattern matching.

Table 1: LongRLVR (bottom rows) shows a massive delta over SFT and Naive RLVR across every scale (7B, 14B).

Critical Insight: Why This Matters

The fundamental takeaway is that Retrieval is Reasoning. We often treat long-context LLMs as having a "big stomach" (memory), but this paper treats them as having "active eyes" (attention). By turning the retrieval of context into a verifiable step in an RL loop, we solve the optimization bottleneck that has plagued long-context models.

Limitations:

1. Synthetic Dependency: The method relies on a high-quality synthetic pipeline to tag "ground-truth chunks," which might be difficult for extremely subjective or open-ended tasks.
2. Output Overhead: The model must output chunk IDs, which adds slightly to the token generation cost.

Future Outlook: This approach could be the key to training "Infinite Context" agents that don't just read everything, but learn exactly what to ignore.