RL for LLM Agents: Capability Expansion or Just Better Reliability?

Executive Summary

TL;DR: Does Reinforcement Learning (RL) actually make LLM agents "smarter," or does it just make them more consistent? A new study introducing the PASS@(k, T) metric reveals that for complex, sequential tool-use tasks, RL genuinely expands the "capability boundary" of agents—solving problems that the base model could never solve, no matter how many times you re-sampled. Interestingly, Supervised Fine-Tuning (SFT) on the same data actually shrank this boundary.

Background Positioning: This work bridges the gap between the pessimistic findings in static reasoning (where RL is seen as a mere "probability redistributor") and the empirical successes of agentic RL (like Agent-R1). It establishes a rigorous framework to prove that in the right environment, RL is a true teacher of new skills.

The Core Conflict: The Two Dimensions of an Agent

Existing metrics like pass@k only measure how many times you try (sampling reliability). But agents have a second dimension: Interaction Depth (T).

The authors argue that a "bridge" question (e.g., "What is the nationality of the director of Film X?") requires two steps. At $T=1$, you can't solve it. No amount of re-sampling ($ko\infty$) helps if you only have one move. You must have depth.

The PASS@(k, T) metric jointly varies these two:

1. k-axis: Reliability. Does the model solve what it knows how to solve?
2. T-axis: Capability. Does deeper interaction unlock new solutions?

Figure 1: The PASS@(k, T) surface. RL (right) shows warmer colors at high depth and sampling on complex tasks compared to the base model.

Methodology: RL vs. SFT

The researchers used Qwen2.5-7B and trained it on three categories:

• A: Pure Math (No tools)
• B: Independent Retrieval (Comparison)
• C: Sequential Retrieval (Bridge questions)

They compared three versions of the model: Base, SFT (on expert paths), and RL (using GRPO). The genius of the setup is "matched data"—SFT and RL saw the exact same 200 problems. Any difference in performance comes from the way they learned, not what they saw.

Key Discovery: The Divergence at the Tail

In static reasoning, researchers previously found that RL and Base models eventually converge as you increase $k$. However, in Category C (Sequential Retrieval), the authors found the opposite:

The gap between RL and the Base model actually widens as you give them more samples.

At $k=1$, the base model was slightly better. But at $k=64$, RL pulled ahead by +4 percentage points. This indicates that RL discovered rare but effective strategies that the base model could almost never find.

Figure 2: On Category C, RL (green) crosses over the Base model and pulls ahead at the high-k limit, proving capability expansion.

The "SFT Trap"

One of the most provocative findings is that SFT regressed the capability boundary. While SFT models often look better at pass@1 (reliability), they "collapse" the model's strategy diversity. SFT effectively told the model: "Do it exactly this way." If that way didn't work for a specific edge case, the model had no other strategies to fall back on. RL, by contrast, preserved strategy diversity while reweighting it toward success.

Deep Insight: How Does it Work?

Through Perplexity Decomposition, the authors found that RL doesn't necessarily learn "magic search queries." Instead, the "novelty" in the RL model's successful runs was concentrated in how it reasoned about the information it found.

RL reweights the base model's existing strategy distribution toward the subset whose downstream reasoning lands on the correct answer.

Figure 3: Perplexity and Diversity analysis. RL stays closer to the base distribution than SFT, but uses those strategies more effectively.

Critical Analysis & Conclusion

The Takeaway: If you are building an agent for simple tasks, SFT is fine. But for compositional, multi-step tasks, skip the expert trajectories and use RL (GRPO). RL's ability to explore and "stitch together" environment feedback allows it to surpass the inherent limits of the base model.

Limitations: The study was limited to a 7B model and a specific retrieval tool. Whether these findings scale to 70B+ models or more "open-ended" tools (like a web browser) remains an open, albeit promising, question.

Future Outlook: PASS@(k, T) should become a standard for agent evaluation. It finally gives developers a way to answer: "Do I need more samples ($k$), or do I need my agent to think deeper ($T$)?"