[ACT] Agentic Critical Training: Moving Beyond Imitation to Genuine Agent Reasoning

TL;DR

Training LLM agents to act like experts is easy; training them to understand why they act is the hard part. Agentic Critical Training (ACT) moves away from simple Imitation Learning (IL) and instead uses Reinforcement Learning (RL) to teach agents how to choose between good and bad actions. The result? Agents that can recover from failures, generalize to unseen environments, and surprisingly, get better at math and physics without even trying.

The "Imitation Trap" in LLM Agents

Most current agents are trained via Imitation Learning (IL)—essentially reading an expert's diary and trying to replicate the same words. The problem? When the environment changes slightly (e.g., an "unseen" room in ALFWorld), the agent becomes a "stochastic parrot" in a loop. It knows what the expert did in that specific room, but it doesn't understand the quality of actions.

Even worse, methods that try to "inject" reflection usually just provide a pre-written explanation for the model to memorize. This is "Imitated Reflection," not "Genuine Reasoning."

Methodology: The Architecture of Choice

ACT fundamentally shifts the learning paradigm. Instead of asking "What is the next token?", it asks "Which of these two actions is better?".

1. Data Construction (Preference Pairs)

For every expert state-action pair, the researchers sample an alternative (and likely suboptimal) action using the base model. This creates a "Contrastive Pair":

• Action A: The Expert Choice (The Gold Standard).
• Action B: The Model Choice (The potentially flawed alternative).

2. The Training Loop (GRPO)

The model is trained using Group Relative Policy Optimization (GRPO). It is presented with the two actions in a randomized order and tasked to:

1. Reason about the situation (Chain-of-Thought).
2. Select the correct action inside <action> tags.

The model is only rewarded for the correct selection. To get that reward consistently, it must learn to "think" about action quality autonomously.

Experimental Results: Performance and Generalization

The authors tested ACT across three benchmarks: ALFWorld (Embodied), WebShop (Web Navigation), and ScienceWorld (Scientific Logic).

SOTA Comparison

As shown in the table below, the "RL w/ ACT" configuration consistently outperformed all baselines, including "Early Experience" (which uses SFT for reflection).

Breaking the "Failure Loop"

One of the most profound findings is ACT's ability to recover from errors. In ALFWorld, when an action fails, standard IL models often repeat the same failure indefinitely (an "infinite loop"). ACT models, having learned to evaluate state/action quality, recognize the failure and pivot to a new strategy (Failure Recovery).

The Reasoning "Side-Effect"

Perhaps the most startling discovery is ACT's effect on general reasoning (MATH and GPQA).

• Imitation Learning (IL) causes "Reasoning Collapse": The model gets so focused on short, action-heavy sequences that it forgets how to do complex algebra or physics.
• ACT actually improves these scores. By training the model to verify actions, it develops a "check your work" habit that translates to solving particle physics problems via self-verification.

Critical Insight & Conclusion

ACT proves that discrimination is a precursor to high-quality generation. By forcing a model to judge action quality through the lens of RL and verifiable rewards, we don't just get better agents; we get more robust, reflective reasoners. The "imitation" era of agent training is likely nearing its end, making way for "critical" RL-driven architectures that understand the why behind the what.

Takeaways for the Future:

• RL over SFT: For complex decision-making, rewarding outcomes is more effective than imitating behaviors.
• Cross-Domain Value: Agentic RL training can be a "backdoor" to improving general-purpose reasoning in LLMs.
• Data Efficiency: ACT data can be transferred across model sizes (e.g., from 8B to 4B) without losing efficacy.