Harmonizing Experts: Why Two-Phase Distillation is the Key to Multi-Task Agentic LLMs

TL;DR

Building a "Jack of all trades" LLM usually leads to a "Master of none" due to task interference. This paper from Tsinghua University and Ant Group reveals that the problem lies in the mode-covering nature of standard distillation. By moving from a single-stage SFT to a Two-Phase Distillation (Off-policy initialization + On-policy refinement), they demonstrate how a single model can finally match the performance of individual task-specific experts.

Context: The Multi-Task Bottleneck

In the quest for AGI, we want LLMs to handle everything from booking flights to solving Sudoku. However, training a single model on multiple tasks via Reinforcement Learning (RL) is notoriously unstable—gradients clash, and one task often "bullies" another out of the weights.

The alternative—training separate experts and then merging them—usually leads to performance trade-offs. The authors identify a fundamental mathematical trap: Forward KL divergence (used in off-policy distillation) tries to cover every behavior the teachers show. When you have multiple teachers, the student's capacity is overwhelmed, leading to a "fuzzy" average policy that fails at higher-level reasoning.

Methodology: The Mode-Seeking Solution

The core insight is the distinction between Forward KL (Mode-covering) and Reverse KL (Mode-seeking).

• Phase 1: Off-Policy Initialization: The student learns the broad strokes of all tasks by mimicking teacher rollouts. This provides the "adequate initialization" needed for the next phase.
• Phase 2: On-Policy Refinement: Once initialized, the model samples its own trajectories. By optimizing the reverse KL (on-policy), the model is incentivized to ignore the "noise" and commit to specific successful reasoning paths (modes).

Figure 1: Comparison of learning curves. Off-policy distillation provides a strong start, but on-policy refinement is what pushes performance back to expert levels.

The Toy Example Intuition

Imagine a model trying to learn two domains simultaneously. If Domain A says "A leads to B" and Domain B says "A leads to C," an off-policy student tries to do both, effectively doing neither. An on-policy student, however, picks the path that provides the highest "reward" (closeness to the expert for that specific input), resolving the conflict.

Experimental Results: Recovering Expert Performance

The authors tested this on τ2-bench (conversational agents) and GEM (reasoning games).

Key Findings:

1. Recovery: In nearly every task, the two-phase approach ("Off. + On.") matched or slightly exceeded the Single-Task RL baseline.
2. Capacity Resilience: This held true for both 8B and 30B models, showing that even larger models suffer from mode-averaging without the refinement phase.
3. Failure of Merging: Traditional parameter merging (like Task Arithmetic) was unstable and sensitive to how the experts were originally fine-tuned.

Table 1: The "Off. + On." column consistently shows the best results, effectively erasing the "Multi-task tax" seen in standard distillation.

Critical Insight & Practical Value

The study proves that data filtering is a viable alternative if on-policy distillation is too expensive. By filtering out tokens that contradict the multi-task model's "high-confidence" modes, they could recover most of the performance lost in standard off-policy training.

Limitations

• Computational Cost: While cheaper than multi-task RL, it still requires running two distinct training phases and hosting multiple teacher experts.
• Expert Dependency: If your single-task experts are mediocre, your multi-task student will be too.

Conclusion

As we push for smaller, more efficient models (slimming down from 400B+ to 8B or 30B), the "student capacity" problem will only grow. This two-phase distillation recipe offers a mathematically grounded and empirically robust framework for consolidating diverse AI capabilities without losing the "expert edge."