[Microsoft Research] Online Experiential Learning: Turning Deployment into a Continuous Training Loop

TL;DR

Most Large Language Models (LLMs) are "frozen" once they leave the training lab. Online Experiential Learning (OEL) breaks this ceiling by allowing models to learn from their own real-world interactions. By extracting structured "knowledge" from textual environment feedback and distilling it into its own parameters, OEL enables a virtuous cycle of improvement in task accuracy and reasoning efficiency—all without human labels or scalar rewards.

Problem & Motivation: The Deployment Paradox

In the current AI lifecycle, deployment is the end of learning. However, the real world is an infinite stream of diverse tasks that no offline dataset can fully cover.

Existing approaches face two major hurdles:

1. Unstructured Feedback: Real-world environments don't provide a "+1 reward" signal; they provide text like "You hit a wall" or "Error: syntax invalid." Standard Reinforcement Learning (RL) cannot handle this.
2. Environment Isolation: Servers where models are trained often cannot access the live user-side environments due to privacy or technical constraints.

OEL solves this by treating the model's own interaction history as a goldmine for autonomous refinement.

Methodology: The Virtuous Cycle of OEL

OEL operates in a recursive loop consisting of two primary stages:

1. Knowledge Extraction (User Side)

The model acts in the environment, collecting trajectories ($T$). Instead of just "remembering" these paths, an extraction model (${\pi }_{extract}$) summarizes them into Experiential Knowledge. This converts a messy sequence of moves into transferable rules (e.g., "Moving into 'H' tiles causes a reset").

2. Knowledge Consolidation (Server Side)

To move this knowledge from the "prompt" (In-Context Learning) into the "brain" (Model Weights), OEL uses On-Policy Context Distillation.

• The student model generates responses.
• A "teacher" (the same model but given the extra Experiential Knowledge in its prompt) guides the student.
• The student is optimized via Reverse KL Divergence, which encourages the model to seek the "modes" of successful behavior.

Experiments: Accuracy Up, Latency Down

The researchers tested OEL on complex text-based games (Frozen Lake, Sokoban) using the Qwen3 family. The results reveal three critical insights:

1. Progressive Improvement

Unlike offline training, OEL's performance doesn't plateau quickly. Each iteration provides higher-quality trajectories, which lead to even better "Knowledge," triggering a fresh wave of improvement.

2. Efficiency Gains (Thinking Faster)

One of the most striking findings is that as the model internalizes experience, its Response Length decreases. It effectively learns to skip "unnecessary thinking" because the rules of the environment are now part of its internal parameters rather than something it has to puzzle out in every prompt.

3. Avoiding Brain Drain (OOD Preservation)

Standard fine-tuning often causes "Catastrophic Forgetting." OEL’s choice of On-Policy training is the secret sauce here. By training on the model's own distribution, it retains its general intelligence (measured by IF-Eval) while mastering the specific environment.

Critical Analysis & Conclusion

The value of OEL lies in its reward-free nature. It bypasses the need for complex Reinforcement Learning from Human Feedback (RLHF) by using the LLM's own summarization capabilities as a "proxy reward."

Key Takeaways:

• Knowledge beats Raw Data: The study proved that distilling extracted knowledge is significantly more effective than distilling raw trajectories.
• Self-Consistency is Key: Knowledge extracted from its own "self" is more useful for a model than knowledge from a larger, smarter model that it cannot emulate.

Limitations: Currently, OEL is validated on text-based environments. Extending this to high-stakes real-world software engineering or physical robotics remains the next frontier.

OEL proves that for LLMs, experience is the best teacher, and deployment is just the beginning of the journey.