TL;DR

The evolution of AI agents is shifting from lone-wolf tool users to coordinated swarms. This paper argues that the traditional RL "trajectory" is dead for multi-agent systems. Instead, we must optimize Orchestration Traces—complex temporal graphs where the most critical decisions are not what tokens to speak, but when to spawn new agents, whom to delegate to, and when to stop.

The "Signal Diffusion" Problem: Why MAS RL is Hard

In a single-agent reasoning task, credit assignment is already difficult. In a multi-agent system, it becomes a nightmare. If a team of 10 agents generates 1,500 steps and eventually finishes a task, how do you reward a single pivotal message sent by agent #3 at step 40?

Under naive shared rewards, the learning signal "diffuses." The paper notes that as trace length $T$ increases, the Signal-to-Noise Ratio (SNR) of any individual decision vanishes. This leads to "free-riding," where lazy agents get credit for doing nothing, or "pseudo-parallelism," where the orchestrator spawns useless agents just to exploit parallelization bonuses.

Methodology: The Anatomy of an Orchestration Trace

The core of this work is the formalization of the Orchestration Trace ($G$) as an event graph. Unlike a linear sequence, $G$ captures:

• Spawn/Despawn Events: The dynamic creation of sub-agents.
• Delegation/Aggregation: How tasks are split and results are fused.
• Message-Level Flows: Causal dependencies between agent utterances.

The paper defines a Dynamic-Dec-POMDP where the state space itself changes as the orchestrator adds or removes vertices in the interaction graph.

The Reward-Credit Dual

The survey provides a brilliant taxonomy of how to tackle the credit assignment problem. It identifies a "dual" relationship:

1. Dense Rewards: Using Process Reward Models (PRM) or verifiers at every turn (e.g., MALT).
2. Structural Credit: Keeping rewards sparse but using mathematical tools like Shapley values (e.g., SHARP) or counterfactual interventions (e.g., C3) to find the "pivotal" event.

Industrial Reality Check: The Scale Gap

One of the most sobering insights is the "Scale Gap." While academic papers brag about 5-agent teams, industrial reports like Kimi K2.6 describe 300-agent swarms with 4,000 steps of coordination.

• Kimi PARL: The first public example of a trained orchestrator using "Critical-Steps" metrics to reward genuine speedup.
• The Harness Constraint: In production (Codex, Claude Code), the model is trapped inside a "Harness." RL must respect the tools and prompts provided by this shell, yet most academic methods still train in bespoke, unrealistic environments.

The Verdict: What's Next?

The paper concludes with 15 open problems. The most glaring? O5: When to stop. We currently have no RL method that explicitly teaches an orchestrator to know when the job is done and stop spending compute.

If we want to reach the next frontier of "Visual Agentic Intelligence," we need benchmarks that measure Parallelism Efficiency ($T_{serial}/T_{parallel}$) and Error Amplification, not just simple accuracy.

Key Takeaways

• Orchestration is a Policy: Spawning an agent is an action that should be rewarded based on its marginal contribution.
• Message is the Unit: We need more research into message-level counterfactuals ($E[R|msg]-E[R|no\_msg]$).
• Systems Matter: Rollout costs for agent swarms are so high that we must adopt "Agentic Pipeline Parallelism" (like MarsRL) just to make training feasible.

This analytical review was compiled based on the "Reinforcement Learning for LLM-based Multi-Agent Systems" survey (May 2026).