SDAR: Taming Multi-Turn Agents with Gated Self-Distillation

TL;DR

Post-training LLM agents often rely on Reinforcement Learning (RL), but coarse rewards make it hard to learn complex, multi-turn interactions. SDAR (Self-Distilled Agentic Reinforcement Learning) solves this by introducing a "gated" distillation process. It uses a privileged teacher (the same model with extra "skills") to provide dense, token-level guidance, but strategically ignores the teacher when it becomes unreliable due to multi-turn drift. The result? A massive performance boost (up to +10%) and a model that successfully "internalizes" expert knowledge.

The Problem: Why OPSD Fails for Agents

In the world of LLM training, On-Policy Self-Distillation (OPSD) is a powerful tool. You take a student model and a "teacher" (the same model but with extra context like gold answers or expert skills). The student learns to match the teacher's better predictions.

However, for multi-turn agents, this breaks down due to two fatal flaws discovered by the authors:

1. Multi-turn Instability: Once an agent makes a small mistake, it drifts away from the teacher's expected path. Naive distillation forces the student to "hallucinate" corrections, leading to exploding KL divergence.
2. Asymmetric Trust: The teacher isn't always right. If the retrieved "skills" are irrelevant or noisy, the teacher might suppress a perfectly good action.

Figure 1: Left - Performance collapse in naive OPSD; Right - Training instability in RLSD-style hybrids.

Methodology: The Sigmoid Gate

Instead of a uniform distillation loss, SDAR treats distillation as a gated auxiliary objective. The primary driver remains RL (using the GRPO algorithm), which ensures the model stays focused on the final environment reward.

The Gating Mechanism

The core innovation is the Teacher-Student Gap Gating. For every token $y_t$, SDAR calculates a gap ${\Delta }_t$: ${\Delta }_t=\log {\pi }_{Teacher}-\log {\pi }_{Student}$

This gap is then passed through a sigmoid function with a sharpness parameter $\beta$: $g_t=\sigma (\beta \cdot {\Delta }_t)$

• Positive Gap (Trust): If the teacher is much more confident about a token the student sampled, $g_t$ approaches 1. This token enters a "strong distillation" regime.
• Negative Gap (Skepticism): If the teacher is less confident (likely due to noise), $g_t$ drops toward 0. The guidance is attenuated, letting the RL reward take the lead.

Figure 2: The SDAR framework architecture showing the dual RL and Gated OPSD pathways.

Experimental Results: Breaking the SOTA

The authors tested SDAR on the Qwen2.5 and Qwen3 series across three rigorous agent benchmarks: ALFWorld (embodied tasks), WebShop (interactive shopping), and Search-QA (multi-hop reasoning).

Key Performance Gains

• ALFWorld: SDAR achieved a 9.4% improvement over vanilla GRPO on the 3B model.
• WebShop: Accuracy jumped by 10.2% for the 7B model.
• Robustness: Even when using randomly retrieved skills, SDAR still outperformed pure RL. This proves the gating mechanism successfully filters out noise and extracts only the "useful" signals from the teacher.

Figure 3: Overall Performance comparison showing SDAR's stability and consistent leads.

Skill Internalization

One of the most impressive findings is that SDAR models do not need the privileged skills during inference. The gated distillation "bakes" the expertise into the model's weights during training. In contrast, other methods like Skill-GRPO drop in performance by nearly 20% if the skills are removed at test time.

Critical Analysis & Conclusion

Why it works

The "Asymmetric Trust" philosophy is brilliant because it acknowledges that "privileged context" is a double-edged sword. By using a smooth sigmoid gate instead of a hard temporal schedule (like TCOD), SDAR allows the model to learn at its own pace, token by token.

Limitations

While SDAR is robust, it still relies on the existence of a "Skill Bank." For entirely new domains where no skills or demonstrations exist, the benefit of the OPSD branch might diminish. Additionally, the hyperparameters $\lambda$ (distillation weight) and $\beta$ (sharpness) require careful tuning to prevent the student from being overwhelmed.

Future Outlook

SDAR provides a blueprint for "Agentic Post-training." As we move toward more complex LLM agents, the ability to selectively distill knowledge from unreliable but "privileged" teachers will be crucial for creating models that are both expert and stable.

Final Takeaway: Don't force your student model to follow every word a teacher says—build a gate and let the rewards prove who was right.