CoPD: Breaking the Specialized Expert Ceiling via Parallel Co-Evolution

TL;DR

Consolidating diverse reasoning skills—like solving complex calculus (text) and interpreting scientific diagrams (image)—into one LLM usually leads to a "jack of all trades, master of none" scenario. Co-Evolving Policy Distillation (CoPD) changes this by training multiple specialized branches in parallel that act as "mutual teachers." By interleaving domain-specific RL with mutual distillation, CoPD ensures that expert knowledge is absorbed while it is still "fresh," allowing a single model to finally outperform individual specialists.

The Problem: The Absorption Gap

In the world of post-training, researchers usually choose between two evils:

1. Mixed RLVR: Throwing all data into one pot. This causes Gradient Conflict, where the optimization direction for "Video Understanding" cancels out the gains for "Mathematical Logic."
2. Static OPD Pipeline: Training experts separately and then distilling them into a student. The authors prove this fails because of the Behavioral Consistency Hypothesis: if a teacher is too advanced or "different" (low Top-k token overlap), the student simply cannot understand the teacher's logic.

Figure: The pilot study confirms that as training progresses, a static teacher's behavior drifts away from the student, leading to a collapse in "absorption efficiency."

Methodology: The "Mutual Teacher" Cycle

CoPD replaces the linear pipeline with a cyclic, parallel architecture. It maintains $K$ branches (e.g., Text, Image, Video) that start from the same base model.

1. Phase I: Branch-Specific RLVR (Divergence)

Each branch uses GRPO (Group Relative Policy Optimization) on its own domain. This "stretches" the branches apart, creating a knowledge gap. This gap is necessary—if they were identical, there would be nothing to teach.

2. Phase II: Mutual OPD (Convergence)

Branches then trade places. The Text branch generates thoughts on Image data, and the Image branch "corrects" it at the token level. Because this happens frequently, their behavioral patterns (Top-K overlap) stay above 0.90, keeping the lessons "absorbable."

Figure: The CoPD framework interleaves exploration (RLVR) and consolidation (OPD) to maintain the "absorbable zone."

Experimental Battleground

The researchers tested CoPD on Qwen3-VL-4B, scaling from two branches (Text/Image) to three (Text/Image/Video).

• Surpassing the Expert: In many benchmarks like AIME (Math) and WeMath (Visual Math), the CoPD-merged model outperformed the domain-specific experts who were only trained on those specific tasks.
• Scaling Victory: In the three-branch setting, CoPD significantly beat Multi-teacher OPD (MOPD). While MOPD struggled to balance three different signals, CoPD’s co-evolution kept all three synergistic.

Figure: CoPD achieves a superior balance across Text and Vision benchmarks compared to both Mixed RLVR and Static OPD.

Critical Insight: The Parallel Self

CoPD is the third part of a "Self-Taught RLVR" series. While previous works looked at the Informed Self (privileged info) and the Temporal Self (future teaching past), CoPD investigates the Parallel Self.

This suggests a future where model scaling isn't just about more parameters or more data, but about architectural orchestration—training a "council" of parallel selves that continuously synchronize their worldviews.

Limitations

• Compute Overhead: Running $K$ branches in parallel significantly increases the VRAM and FLOPs required during the training phase.
• Synchronization Rhythm: The ratio between exploration (RLVR) and consolidation (OPD) is sensitive (found to be optimal at 1.5:1). Finding this "sweet spot" for 10+ capabilities might be challenging.

Conclusion

CoPD effectively proves that "Distance makes the heart grow fonder" does not apply to AI distillation. By keeping specialized branches behaviorally close through mutual co-evolution, we can finally build unified models that are truly the sum of their expert parts.