ReOPD: Escaping the Prefix Trap in Multi-Turn Agent Distillation

TL;DR

On-policy distillation (OPD) is the "gold standard" for teaching small models to reason like giants, but its cost—driven by real-time environment interaction—is a massive bottleneck. ReOPD (Replayed-Prefix On-Policy Distillation) eliminates this cost by replaying offline teacher traces while preserving the "on-policy" signal. By introducing a reliability-aware decay schedule, it not only trains 4x faster but actually outperforms standard online distillation in complex reasoning tasks.

The Problem: The High Cost of Interaction & The "Prefix Trap"

Training an LLM agent (e.g., a model using a Python interpreter or a search engine) usually requires the model to interact with the environment during training.

1. Cost: Online interaction requires running sandboxed code or calling APIs thousands of times per training step.
2. The Prefix Trap: Standard OPD suggests the student should generate its own history to stay "on-policy." However, if a student makes a weird mistake early on, it enters a state the teacher doesn't understand. If you ask a PhD-level "Teacher" model to help a "Student" who has already made a nonsensical calculation, the teacher's advice might become unreliable. This is the Teacher Reliability Shift.

Methodology: Reliability-Aware Prefix Design

Instead of fresh rollouts, ReOPD reuses trajectories the teacher already generated during its own RL training.

1. Teacher-Forced Prefix, Student-On-Policy Action

For any given step $t$, ReOPD takes the teacher's history (Prefix) and gives it to the student. The student then generates its own action. We then calculate the loss by comparing the student's output to the teacher's distribution. This keeps the environment "frozen" (offline) while the training remains "on-policy" at the specific moment of decision.

2. The Step-Decay Schedule

As the trajectory gets longer, the gap between what the teacher did and what the student would have done grows. ReOPD solves this by using a step-decaying sampling weight ($w_t={\kappa }^t$). It emphasizes training on earlier turns where the teacher-student alignment is high and the teacher's guidance is most trustworthy.

Experimental Results: Faster and Sharper

The authors tested ReOPD across mathematical reasoning (Python tool-use) and Search/QA tasks using the Qwen3 family.

• Math Reasoning: In the AIME24 benchmark, ReOPD improved the student's score from 35.4% (standard OPD) to 40.8%.
• Efficiency: Because ReOPD needs zero live tool calls, it is at least 4x faster and significantly reduces GPU/memory overhead required for environment deployment.

Deep Insight: Why Not Just Go Fully On-Policy?

The most striking finding of this paper is that "Fully Student-On-Policy" is not always optimal. When the gap between the teacher and student is large (e.g., a 30B teacher vs. a 4B student), the student is so likely to wander off into "nonsense" states that the teacher's supervision becomes noise. By "anchoring" the student to teacher prefixes and decaying the loss weight for later steps, ReOPD finds a "Geometric Bridge" that maximizes the density of the learning signal.

Conclusion & Future Outlook

ReOPD turns the expensive "online" requirement of agent training into a "reusable offline resource." This is a major step toward scaling small, efficient models that can use tools as effectively as frontier models. The next frontier will likely be adaptive scheduling, where the model learns exactly when to trust the teacher and when to forge its own path.

Takeaway: If your teacher is much smarter than your student, don't let the student lead the way—replay the teacher's path, but let the student take the wheel at every corner.