SDAR: Taming Multi-Turn Agents with Gated Self-Distillation

TL;DR

Training LLM agents to handle multi-turn interactions is notoriously difficult due to "sparse" trajectory rewards and the instability of token-level supervision. This paper presents SDAR (Self-Distilled Agentic Reinforcement Learning), a novel framework that keeps the unbiased nature of Reinforcement Learning (RL) while selectively injecting "privileged knowledge" via an adaptive token-level gate. By trusting the teacher only when it provides positive endorsements, SDAR shatters the stability bottleneck of prior hybrid RL-distillation methods.

The Core Conflict: RL vs. Distillation

In the quest to create autonomous agents, researchers typically use two tools:

1. Reinforcement Learning (RL): Grounded in environment feedback, but provides "coarse" signals (success/failure) only at the end of a long sequence.
2. On-Policy Self-Distillation (OPSD): Provides dense "token-level" guidance by comparing the agent to a "privileged teacher" (the same model given extra hints/skills).

The Problem? In multi-turn settings, if a student agent makes a small mistake early on, it drifts away from the teacher's path. Naive distillation then forces the student to follow a teacher that might be equally "lost" due to the drift, leading to catastrophic performance collapse.

Intuition: Asymmetric Trust

The authors made a profound observation: The Teacher is not always right. If the teacher (with access to skills) assigns a higher probability to a token than the student, it’s a "positive endorsement"—the skill helped. However, if the teacher assigns a lower probability, it might be due to a "negative rejection" caused by bad skill retrieval or noise.

SDAR addresses this by treating OPSD as a gated auxiliary objective. Instead of forcing the student to match the teacher everywhere, it uses a Sigmoid Gate to decide how much to distill each token.

Figure 1: The SDAR framework combines trajectory-level GRPO rewards with token-level gated distillation.

Methodology: The Token-Level Gate

The mathematical secret of SDAR lies in the Teacher-Student log-probability gap (${\Delta }_t$).

${\Delta }_t=\log {\pi }_T(y_t|s_t^{+})-\log {\pi }_{h}eta(y_t|s_t)$

SDAR maps this gap through a sigmoid function to create a gate ($g_t$):

• Positive Gap: Teacher is more confident $\to$ Gate opens wide (Distill strongly).
• Negative Gap: Teacher is less confident $\to$ Gate closes (Attenuate signal).

This creates a "self-paced curriculum." Early in training, the gate filters out noise; as the student improves, the distillation becomes more surgical.

Experimental Performance

The researchers tested SDAR on the Qwen2.5 and Qwen3 families across three rugged benchmarks: ALFWorld (embodied tasks), WebShop (online shopping), and Search-QA.

Key Breakthroughs:

• Stability: While naive hybrid methods (GRPO+OPSD) often crash on small models (like 1.7B), SDAR remains remarkably stable.
• Internalization: Unlike agents that need "hints" at test time, SDAR agents learn to internalize the skills into their weights. It even outperformed models that had access to skills at test time.
• Robustness: Even when the skill retrieval was randomized (noise), the gating mechanism was smart enough to ignore the garbage, still outperforming pure RL.

Figure 2: Performance gains of SDAR over standard GRPO.

Critical Analysis: Why This Matters

The most striking takeaway is the Gate Activation Ratio. In early training, the gate is mostly closed (ratio < 0.5), protecting the agent from misleading teacher signals. As the policy evolves, the gate opens, allowing the teacher to "fine-tune" the nuances of the agent's reasoning.

Limitations: While SDAR is robust to retrieval quality, the choice of the sharpness parameter ($\beta$) and the distillation weight ($\lambda$) still requires calibration. If $\lambda$ is too high, the teacher's "noise" can still overwhelm the RL reward.

Conclusion

SDAR represents a shift in thinking for LLM post-training. By moving away from "uniform distillation" and toward "asymmetric trust," we can finally leverage privileged context to train multi-turn agents that are both powerful and stable. It proves that in the world of AI agents, knowing when not to listen to your teacher is just as important as the lesson itself.