[MSR 2026] Online Experiential Learning: Turning Deployment into a Continuous Training Loop

TL;DR

Microsoft Research proposes Online Experiential Learning (OEL), a framework that breaks the "static deployment" barrier. Instead of treating production interactions as discarded data, OEL extracts insights from textual environment feedback and redistills them into the model's weights. The result? A model that gets smarter and faster the more it is used, without needing humans to label data or engineers to design complex reward functions.

The "Static Model" Problem

Traditionally, LLM development follows a "Train then Deploy" waterfall. Once a model is out in the wild, its intelligence is frozen. If it encounters a new task or a peculiar environment, it cannot adapt. The reason is simple: standard Reinforcement Learning (RL) requires scalar rewards (numbers), but real-world interactions provide textual feedback (e.g., "Error: Box hit the wall"). Designing a reward model for every user scenario is impossible.

Authors argue that we are wasting a "virtually unlimited source of learning signal" by not utilizing these textual trajectories.

Methodology: The Virtuous Cycle of OEL

OEL operates through a recursive loop of Extraction and Consolidation.

1. Stage One: Knowledge Extraction

The model interacts with the environment (e.g., a text game or a user task) and generates a trajectory. A "knowledge extractor" (often the model itself) then analyzes these trajectories to identify what worked and what didn't. This is converted into Experiential Knowledge—a set of "Experience Items" that describe transferable rules or strategies.

2. Stage Two: On-Policy Consolidation

How do you get this knowledge into the model's "brain" without making it reliant on long prompts? OEL uses On-Policy Context Distillation.

• Teacher: The model + the extracted knowledge.
• Student: The raw model. The student tries to predict what the "knowledgeable" teacher would do. By using Reverse KL Divergence, the student internalizes the teacher's expertise into its own weights.

Experimental Validation

The authors tested OEL on text-based puzzle environments like Sokoban and Frozen Lake.

Continuous Improvement

As shown in the results below, the "Pass Rate" doesn't just jump once; it climbs steadily with each iteration. Each round of learning provides a better baseline for the next round of exploration.

Thinking Faster, Not Just Better

Interestingly, OEL also improves Inference Efficiency. As the model internalizes the knowledge, it requires fewer "thought tokens" to arrive at the correct answer. In Frozen Lake tests, the response length dropped to 70% of the original while accuracy increased.

Why On-Policy Matters

A critical insight in this paper is the comparison between On-Policy vs. Off-Policy distillation. Standard off-policy training (matching a pre-generated expert dataset) often leads to Catastrophic Forgetting—the model gets better at the specific task but becomes "dumber" at general instructions (OOD tasks).

OEL’s on-policy approach forces the model to learn from its own potential mistakes, which significantly preserves its general capabilities (measured by IF-Eval).

Conclusion & Future Outlook

OEL represents a major step toward Autonomous Agents. It proves that the "Era of Experience" is here: models no longer need to be spoon-fed by human annotators. By simply "living" in an environment and reflecting on its failures and successes, an LLM can self-evolve.

The future of AI may not be about "Bigger Datasets," but about "Better Experiences" and the efficiency with which a model can internalize them.

Limitations

• The extraction relies on the model's existing reasoning capability; very weak models might struggle to distill useful knowledge.
• Currently tested on text-based games; scaling to complex real-world software workflows is the next frontier.