[ArXiv 2026] HACRL: Breaking the Silos of LLM Training via Heterogeneous Collaboration

Executive Summary

TL;DR: Heterogeneous Agent Collaborative Reinforcement Learning (HACRL) is a new training paradigm that allows different LLMs (e.g., Llama 3B and Qwen 4B) to train together by sharing their reasoning rollouts. By introducing HACPO, the authors solve the stability issues of learning from "foreign" models, achieving significant performance gains in mathematical reasoning with 50% less sampling cost.

Background: This work sits at the intersection of Multi-Agent RL (MARL) and RL with Verifiable Rewards (RLVR). Unlike traditional MARL where agents coordinate at inference, HACRL agents are independent at deployment but "crowdsource" their experiences during training to break performance ceilings.

1. The Core Motivation: Why Train Together?

In modern LLM development, we often train multiple models (Base, Instruct, Small, Large) for different tasks. Traditionally, each undergoes its own RL process:

1. Sample a trajectory (expensive).
2. Get a reward (e.g., unit test).
3. Update policy.
4. Discard the data.

This is wasteful. Authors argue that a Qwen model can benefit from seeing how a Llama model "failed" or "succeeded" on the same math problem. However, you can't just mix their data—different models have different "baselines" for what a good reward looks like, and their probability distributions (tokenizers/architectures) are vastly different.

2. Methodology: How HACPO Tames Heterogeneity

The authors identify three types of heterogeneity: State (same model, different weights), Size (same family, different parameters), and Model (totally different architectures). To make collaboration work, HACPO introduces four key mechanisms:

A. Agent-Capability-Aware Advantage Estimation

In algorithms like GRPO, rewards are normalized within a group. HACPO adjusts this baseline by a Capability Ratio ($\omega$). This ensures that when a 70B model looks at a 7B model's rollout, it doesn't get "over-excited" by a simple success that the 70B model finds trivial.

B. Exponential Importance Sampling

When learning from another model's rollout, the importance sampling ratio can be volatile. HACPO uses a sequence-level ratio with an exponential decay factor ($\alpha$): $ilde{s}_{t,i}^{(k,j)}=s_{t,i}^{(k,j)}\cdot (extsg[s_{t,i}^{(k,j)}]{)}^{\alpha }$ This forces the model to be conservative when the "foreign" rollout is too far from its own distribution.

C. Stepwise Clipping

Stability is the enemy of multi-agent training. HACRL observes that "foreign" data distribution shifts unpredictably during a training step. HACPO introduces an asymmetric, stepwise clipping strategy that gets stricter as the training batch progresses, preventing the model from being "derailed" by external gradients.

Figure: The four algorithmic pillars of HACPO ensuring stable cross-agent learning.

3. Experimental Results: The Bidirectional Boost

The authors tested HACPO across several benchmarks (MATH, GSM8K, AIME). The results challenge the traditional "Teacher-Student" dogma.

Key Insight: Weak Agents help Strong Agents. Even when pairing a "base" model with an "instruct" model, the base model provides diverse, creative exploration paths (and informative errors) that the more "rigid" instruct model can learn from to improve its robustness.

Figure: Comparison of HACPO vs. Single-agent baseline. Note the faster convergence and higher ceiling.

4. Critical Analysis & Conclusion

Takeaway

HACRL shifts the paradigm from isolated optimization to collaborative ecosystems. It proves that model heterogeneity is a feature, not a bug—different architectures have different inductive biases, and sharing their "worldviews" during RLVR training leads to a more robust global policy.

Limitations

1. Communication Overhead: While it saves on sampling, the gradient updates for multiple models simultaneously require high GPU memory.
2. Tokenizer Alignment: For cross-model collaboration (e.g., Qwen and Llama), "detokenization" and "retokenization" are required, which adds a small computational layer.

Future Work

The authors suggest this could extend to Multi-Modal agents, where a vision model and a language model share reasoning traces to align their internal logic more effectively.

Final Verdict: A must-read for anyone building LLM clusters. HACPO turns the "costly sampling" curse of RL into a collaborative advantage.