CoPD: Breaking the Expert Ceiling via Mutual Co-Evolution

TL;DR

Integrating multiple specialized capabilities (text, image, video reasoning) into a single LLM usually results in a "trade-off" where one skill erodes the other. Co-Evolving Policy Distillation (CoPD) solves this by training multiple "expert branches" in parallel. By alternating between reinforcement learning (to gain new knowledge) and mutual distillation (to keep the experts' "thinking patterns" aligned), CoPD creates a unified model that—for the first time—actually outperforms its specialized teachers.

Thinking Alike: The Behavioral Consistency Hypothesis

Why does standard distillation fail once an expert is "too smart"? The authors propose a Behavioral Consistency Hypothesis: A student can only absorb a teacher's knowledge if they occupy similar states in the latent space.

If a text expert and an image expert are trained to completion separately, they drift apart. By the time you try to distill them together, their token-level distributions are so far apart (measured by Top-k overlap) that the student cannot bridge the gap.

Figure 1: Pilot study proving that as behavioral overlap (Top-k) increases, the effectiveness (Gain) of distillation rises linearly.

Methodology: The Co-Evolution Dance

CoPD replaces the static "Expert -> Student" pipeline with a dynamic Hub-and-Spoke or Pairwise parallel training process. The workflow consists of two tightly coupled phases:

1. Phase I: Branch-Specific RLVR: Each branch (e.g., Text Branch, Image Branch) runs its own Reinforcement Learning (GRPO) on domain-specific data. This "pushes" the branches apart, exploring the frontier of expertise and creating informative differences.
2. Phase II: Mutual On-Policy Distillation (OPD): Immediately after, the branches swap roles. The Text branch teaches the Image branch how to reason through text-based prompts, and vice versa. This "pulls" the branches back together, ensuring their behavioral patterns remain synchronized.

Figure 2: The CoPD loop. RLVR creates expert knowledge; Mutual OPD ensures that knowledge is shared while it is still "fresh" and absorbable.

Results: Surpassing the Experts

The most striking result is the "All-in-One" consolidation. In a 3-branch setting (Text, Image, Video), CoPD achieved:

• Text Reasoning: 58.76% (Higher than the specialized Text-Expert's 57.89%)
• Image Reasoning: 56.97% (Higher than the Image-Expert's 55.76%)

Unlike "Mixed RLVR" (simply training on all data at once), which saw performance drops in text math tasks, CoPD maintains high specialized skill across all domains. This suggests that Model Parallelism isn't just for efficiency—it's a superior training paradigm for multi-modal reasoning.

Table 1: CoPD consistently beats independent experts and standard multi-teacher distillation (MOPD).

Deep Insights: The Rhythm of Learning

The authors investigated the ratio of Exploration ($S_{RL}$) to Consolidation ($S_{OPD}$). They found a "Goldilocks zone" (ratio of 1.5:1).

• Too much RL? The experts drift too far apart (High KL divergence), and distillation fails.
• Too much OPD? The experts become identical too quickly, and there is no new "frontier" knowledge to share.

Figure 3: CoPD maintains Top-k overlap > 0.9 throughout training, whereas the static baseline drops to nearly 0.7.

Conclusion & Perspective

CoPD proves that the "Parallel Self"—multiple instances of a model learning different things and teaching each other simultaneously—is a powerful way to scale intelligence. This sidesteps the fundamental gradient conflict in multi-task learning by allowing models to specialize before they synchronize. For future AI developers, this provides a blueprint for building "Swiss Army Knife" models that don't lose the sharpness of a specialized blade.