MOPD: Shattering the Trade-off in Multi-Domain LLM Integration

TL;DR

Integrating specialized skills (like elite mathematical reasoning and complex software engineering) into a single LLM often results in a "jack of all trades, master of none" scenario. MOPD (Multi-Teacher On-Policy Distillation) breaks this curse. By training specialized teachers in parallel and distilling their wisdom into a student model using the student's own generated paths, MOPD captures nearly 94% of specialized teacher performance—surpassing traditional joint training and weight merging.

The "See-Saw" Problem in Post-Training

In the current LLM landscape, we know how to make a model great at one thing using Reinforcement Learning (RL). We use verifiable rewards for Math and sandboxed execution for Code. However, when we try to merge these capabilities into a single model, we hit a wall:

• Mix-RL: Training on everything at once causes "gradient interference." Improving Math often degrades Instruction Following.
• Cascade RL: Training sequentially leads to the model forgetting the first skill by the time it learns the third.
• Param-Merge: Simply averaging the weights of a Math-LLM and a Code-LLM often results in a model that is worse than both.

MOPD: Integration in Policy Space

The core insight of MOPD is that capability integration should happen during the distillation process, not in the weight space or the raw data pool.

The Three-Stage Architecture

1. Stage 1 (General SFT): Establish a shared foundation.
2. Stage 2 (Parallel Specialists): Branch off and train independent RL teachers for each domain. This allows teams to iterate on Math or Code independently.
3. Stage 3 (On-Policy Distillation): This is the "secret sauce." The student model generates its own answers (rollouts). These answers are sent to the relevant "Teacher Service" which provides token-level feedback (log-probabilities).

Why It Works: Dense Supervision

Unlike standard RL, which often gives a single "reward" at the very end of a long solution, MOPD provides dense optimization signals. The student gets feedback on every single token by comparing its probability distribution to the teacher's. This reduces variance and makes the training far more sample-efficient.

Experimental Results: Efficiency Meets Performance

MOPD was tested against a battery of baselines on the Qwen3-30B architecture.

• Superior Recovery: MOPD achieved a normalized score of 0.937, meaning it inherited almost the entire capability of the specialized teachers.
• Unrivaled Efficiency: As shown in the training dynamics below, MOPD (orange line) hits the performance ceiling much faster than Mix-RL (blue line), purely because the teacher's token-level guidance is so much richer than a simple reward signal.

Critical Insight: The "Same-Origin" Necessity

One of the most profound findings in the paper is that the teacher must be a "same-origin" relative of the student. The authors tried using a much stronger, larger external model (Qwen3-235B) as a teacher. It failed. Because the large teacher's "thought process" (probability distribution) was too different from the student's, the KL-divergence was too high, leading to optimization collapse. Distillation works best when the teacher is a "better version of yourself," not a total stranger.

Conclusion and Future Outlook

MOPD offers a pragmatic, industrial-grade solution to the complexity of LLM post-training. By decoupling "capability production" from "capability integration," it allows large AI labs to develop features in parallel and fuse them stably.

Takeaway for Practitioners: If you are struggling with model degradation while trying to add new skills to your LLM, stop focusing on merging weights. Instead, deploy your specialists as inference services and use on-policy distillation to "teach" your core model how to emulate their outputs.