[Alibaba & HKUST] Complementary RL: How Co-Evolving "Experience" and "Policy" Solves Agent Sample Inefficiency

TL;DR

Reinforcement Learning (RL) for LLM agents is notoriously slow because models often "forget" the procedural nuances of past successes and failures. Complementary RL introduces a bio-inspired architecture where an Experience Extractor (the "Hippocampus") and a Policy Actor (the "Neocortex") are trained simultaneously. By making the experience-gathering process a trainable RL task itself, the system ensures that distilled knowledge stays relevant as the agent grows stronger, leading to a 10% performance jump and 2x higher efficiency in complex environments.

The Stagnation of Static Memory

Standard RL for agents relies on sparse, binary rewards (e.g., "Did you buy the right shoes?"). This ignores the rich "how-to" data hidden in the trajectory. While researchers have tried adding "Experience Banks," these banks usually act like a dusty library:

1. Misalignment: As the agent learns, it needs more sophisticated advice. Static libraries keep giving "beginner" advice to an "expert" agent.
2. Internalization Failures: Agents often become over-reliant on external prompts, failing to bake that wisdom into their own weights.
3. Retrieval Noise: Static extractors often hallucinate or provide irrelevant context, distracting the agent during critical decision-making.

Methodology: The Co-Evolutionary Dual-Loop

The core contribution of this work is the transition from Static Experience to Co-Evolutionary Experience.

1. The Neuroscience Inspiration

The authors leverage the Complementary Learning Systems (CLS) framework. In humans, the hippocampus quickly captures new experiences, which are then "consolidated" into the neocortex for long-term structured knowledge.

2. Dual-Model Architecture

• The Actor (${\pi }_{h}eta$): Interacts with the environment. It is trained using a modified GRPO (Group Relative Policy Optimization). To prevent over-reliance, the authors use a "Split-Group" strategy: half the training samples use experience context, while half do not. This forces the model to internalize the logic.
• The Experience Extractor (${\pi }_{\varphi }$): Distills trajectories into structured textual rules. It is optimized via CISPO based on a "meta-reward": If the experience I generated helped the actor win, I get a positive reward.

3. The "ExperienceManager" Infrastructure

Managing a dynamic memory bank at scale is a DevOps nightmare. The authors designed an asynchronous framework where a central ExperienceManager handles:

• Producer-Consumer Distillation: Trajectories are queued so that the extractor never blocks the actor's interaction.
• Periodic Merging: A background process merges or deletes redundant/conflicting memories to keep the search space efficient.
• Search-and-Ask: This allows the actor to proactively query the memory bank during a task, not just at the start.

Experimental Results: Faster, Smarter, Leaner

The team tested the system against massive benchmarks: MiniHack (Navigation), WebShop (E-commerce), ALFWorld (Household tasks), and SWE-Bench (Software Engineering).

Performance and Efficiency

• Success Rates: Complementary RL outperformed "outcome-only" RL by ~10% and significantly beat "Static Experience" baselines.
• Action Purity: In ALFWorld, the agent completed tasks using 2x fewer actions. It wasn't just succeeding; it was finding the "optimal path" because the evolved experience provided better heuristics.

The "Internalization" Proof

One of the most impressive findings was the Eval w/o Experience results (Table 1). Even when the memory bank was turned off during testing, the Complementary RL agent performed better than the baseline. This proves the co-evolutionary loop successfully "taught" the knowledge to the actor's core parameters.

Depth Insight: Why "Split-Group" Advantage Matters?

A technical "gotcha" found by the authors was that mixing experience-guided and experience-free rollouts in a standard GRPO buffer causes training collapse. Because the rewards for experience-guided runs are naturally higher, the advantage calculation becomes biased. The solution? Condition-wise Advantage Estimation. By calculating the mean and standard deviation separately for the two groups, the gradient signals remain clean, allowing the model to learn from both contexts simultaneously.

Limitations & Future Work

While powerful, the system adds complexity:

1. Model Divergence: In some experiments (like Self-Distillation), the training collapsed late in the process, suggesting the "internalization" part of the loop needs more robust regularization.
2. Compute Overhead: Although the framework is asynchronous, you still need to run two separate training loops for two models, which increases the VRAM footprint.

Conclusion

Complementary RL marks a shift in how we think about "Memory" in AI. It isn't just a database to be queried; it's a living system that must learn what to remember as the agent learns what to do. For developers building agentic workflows, the takeaway is clear: don't just prompt your agent with old success stories—build a system that learns which stories are worth telling.

For more technical details, check out the official repository.