[ICLR 2025 Submission] Beyond In-Domain Success: Does Reinforcement Fine-Tuning Actually Build Generalizable LLM Agents?

Executive Summary

Reinforcement Fine-Tuning (RFT) has become the de facto standard for elevating LLMs from passive text generators to active agents. However, a critical question remains: are these agents learning to "reason," or are they just learning the "interface"?

This study provides a comprehensive audit of RFT generalization across three dimensions: task difficulty, cross-environment transfer, and sequential learning. While the results confirm that RFT is a powerhouse for boosting performance within a known world (Intra-environment), it reveals a fragile reality when the agent is pushed into the unknown (Inter-environment).

The "In-Domain" Illusion

Most current SOTA agents are trained and tested on the same platforms (e.g., specific web-shopping simulators). The authors argue that this creates a false sense of progress. In reality, a robust agent should handle a new website or a new game engine without complete retraining. The gap between Held-In (same environment) and Held-Out (new environment) performance is the true measure of "Agentic Intelligence."

Methodology: The Three Axes of Generalization

The researchers utilized GRPO (Group Relative Policy Optimization), an RL algorithm that optimizes policies without a critic network, reducing overhead while focusing on trajectory-level rewards.

1. Task-Level (Easy vs. Hard): Does learning to buy a "white t-shirt" help you buy a "complex polyester-spandex blend within a specific price range"?
2. Environment-Level: Does training on a search engine (SearchQA) help an agent navigate an e-commerce site (WebShop)?
3. Sequential Axis: Can an agent learn 5 different environments in a row without forgetting how to handle the first one?

Key Insight 1: Reach High by Starting Low

The study confirms that Curriculum Learning (Easy $\to$ Hard) is superior to training on a random mixture of tasks. By mastering simpler interactions first, the agent builds a foundational policy that RFT then refines for the "noisier" signals found in difficult, long-horizon tasks.

Key Insight 2: The Interface Trap

This is the most provocative finding: Specific environment features can act as "crutches." In the BabyAI environment, the system explicitly lists "Available Actions" at every step. Agents trained here became "addicts" to this list. When moved to WebShop (which provides raw HTML instead of action lists), these agents failed miserably, dropping far below the performance of the baseline "un-tuned" model.

Conversely, SearchQA acted as a "gym" for information extraction. Agents trained there learned to refine search queries, a skill that transferred beautifully to WebShop, helping the agent filter through messy HTML metadata.

Key Insight 3: Sequential Training is surprisingly Stable

Modern LLMs are often prone to "Catastrophic Forgetting." However, the authors found that if you train an agent on Environment A then Environment B, it retains most of its A-skills.

• Finding: Sequential RFT performs nearly as well as Joint Training (mixing all data at once).
• Why?: RFT optimizes trajectory-level objectives which seem to preserve the underlying representations better than the "token-matching" nature of Supervised Fine-Tuning (SFT).

Case Study: SearchQA $\to$ WebShop

Instead of blindly pasting the entire user prompt into a search bar (the Base Model's mistake), the RFT-trained agent learned to distill the prompt into core attributes (e.g., material, color, size). This "Information Foraging" behavior is the hallmark of a successful agent.

Conclusion & Future Look

The study concludes that RFT is excellent for "behavioral tightening"—making an agent faster, more efficient, and more logical within a specific domain. However, to build a "Universal Agent," we cannot rely on RFT in a vacuum. We must expose models to a diverse mixture of interfaces to prevent them from overfitting to the "UI" of a specific environment.

Critical Takeaway: The next frontier for LLM agents isn't just more RL steps—it's more diverse RL environments.