Coordination as an Architectural Layer: Predicting Failure in Multi-Agent LLMs
Coordination as an Architectural Layer for LLM-Based Multi-Agent Systems
Summary
Problem
Method
Results
Takeaways
Abstract
This paper proposes treating "Coordination" as a configurable architectural layer in multi-agent LLM systems, distinct from agent logic and information access. Using an information-controlled experimental design on a prediction-market testbed, it demonstrates that specific coordination topologies (e.g., Sequential Pipeline, Consensus Alignment) produce predictable and measurable "failure-mode signatures" using Murphy decomposition.
## TL;DR
Why do sophisticated multi-agent systems often perform worse than simple single-agent baselines? This paper argues that the problem lies in the **Coordination Layer**. By isolating coordination from information access, the authors prove that your choice of how agents talk (topology) creates specific, predictable failure signatures. Using prediction markets as a "ground-truth" lab, they found that common patterns like *Orchestrator-Specialist* are often inefficient, while *Sequential Pipelines* and *Independent Ensembles* define the current Pareto frontier of cost and accuracy.
## The Problem: The "Information-Architecture" Confound
In the current "Gold Rush" of agentic frameworks (like AutoGen or LangGraph), we often see claims that "Multi-agent systems improve reasoning." However, the authors point out a massive methodological flaw: most studies give multi-agent systems more tools, more tokens, and more context than their single-agent baselines.
Is the improvement coming from the **coordination** (the "teamwork") or simply from the **information** (the "extra books")?
According to the paper, most failures (41%-87%) in production agents are "coordination defects"—misalignment, specification ambiguity, and verification gaps—rather than the base model's stupidity.
## The Methodology: Scientific Isolation
The authors treat the system as three distinct layers:
1. **Information Layer**: Tools, data, and context.
2. **Coordination Layer**: Who talks to whom, who decides, and how results aggregate.
3. **Agent Layer**: The underlying LLM (e.g., Claude Opus).
To prove coordination matters, they kept the Information and Agent layers *identical* across five different "topologies" and tested them on **Polymarket** binary predictions. Because markets have absolute outcomes, they provide a "contamination-resistant" way to measure true reasoning.
## The Five Archetypes and Their Signatures
The core of the paper is the **Murphy Decomposition**, which splits the Brier Score (accuracy) into:
* **Reliability (REL)**: Calibration error—how well do "80% confident" predictions actually match reality?
* **Resolution (RES)**: Discriminative power—how well can the model tell a "Yes" from a "No"?
| Configuration | Predicted Signature | Observed Reality |
| :--- | :--- | :--- |
| **Independent Ensemble** | High Diversity/RES | **Frontier.** Best for cost-adjusted accuracy. |
| **Peer-Critique Debate** | Improved Calibration | **Disappointing.** Alignment pressure suppressed dissent. |
| **Orchestrator-Specialist**| High Specialization | **Dominated.** Expensive and prone to "cascading errors." |
| **Sequential Pipeline** | Best Case Power | **SOTA.** Best overall accuracy, though expensive. |
| **Consensus Alignment** | Low Diversity | **Failure.** Collapsed into "confidentially wrong" midpoints. |
## Key Insights: The Pareto Frontier
The most striking result is the **Cost-Quality Analysis**. The researchers plotted how much each "answer" cost in API tokens versus its accuracy.
* **The Winners**: *Independent Ensembles* are cheap and robust. *Sequential Pipelines* are the most accurate if you can afford the 3.6x cost.
* **The Losers**: *Orchestrator-Specialist* and *Peer-Critique* patterns—the darlings of many current AI tutorials—were "dominated." They cost more than the ensemble while delivering worse results.
## Why it Matters: Deep Insights
1. **Convergence is a Trap**: In the *Consensus Alignment* model, agents iterate until they agree. This sounds "safe," but the study showed it actually causes "Diversity Collapse." The agents anchor on a mediocre midpoint, losing the "Wisdom of the Crowd."
2. **Cascades are Deadly**: In *Sequential Pipelines*, if the first "Research" agent makes a slight framing error, the following agents (Analyst, Forecaster) just amplify that error. There is no architectural way to go back and fix the foundation.
3. **Specialization Inflation**: The *Orchestrator-Specialist* model (very common in industrial agents) often fails because the "Orchestrator" becomes a single point of failure. If it decomposes the problem poorly, the specialists provide "perfect answers to the wrong questions."
## Conclusion: Engineering vs. Architecture
This paper is a wake-up call for AI engineers. We possess the tools to build complex "agent swarms," but we are currently doing so without an architectural compass.
**Takeaway**: Stop building "Consensus" loops unless you want to kill your model's intelligence. If you need accuracy, spend tokens on a deep *Sequential Pipeline*. If you need efficiency, a simple *Independent Ensemble* is surprisingly hard to beat.
## Critical Analysis & Limitations
* **Statistically Suggestive**: With $n=100$ markets, the differences between some configurations are not yet "statistically significant" under strict Bonferroni correction. A larger $n=500$ study is needed.
* **Single Model**: The results might change with GPT-5 or localized models.
* **Future Scope**: The authors are running a "Live" replication on the **Foresight Arena** (on-chain) to see if these signatures hold up when web search is enabled.