What do coordination benchmarks actually miss?
Most coordination benchmarks are too short, too structured, or too single-agent to reveal how LLMs handle long-horizon, open-ended teamwork. A 2026 benchmark called alem was built specifically to test these demands together—long-horizon survival with exploration, crafting, trading, and combat—and found that current LLM agents average only ~6% normalized return, meaning they solve only a tiny fraction of the task's potential [2]. This is a stark contrast to typical short-horizon coordination games where LLMs can appear competent [5].
The gap is not just about task difficulty; it's about what the benchmark measures. In alem, a frontier model (GPT-5.4-High) achieved strong base-task reward but much lower coordination reward, showing that a model can be individually skilled yet fail at coordination [2]. This directly contradicts the assumption that better single-agent abilities translate to better teamwork—a point also echoed in a 2023 benchmark where LLMs excelled when decisions relied on environmental variables but struggled when they had to reason about partners' beliefs and intentions [5].
Why do LLMs fail at coordination even when they seem to succeed?
Coordination tasks often fail to expose the underlying reasoning gaps because LLMs can appear to coordinate by pattern-matching environmental cues rather than by genuinely modeling others' minds. In the 2023 LLM-Coordination benchmark, LLM agents performed well in pure coordination games where the right move was obvious from the environment, but their performance dropped in scenarios requiring active consideration of partners' beliefs and intentions [5]. This suggests that benchmarks that don't explicitly test Theory of Mind (ToM) reasoning—the ability to infer what others know and plan—will overestimate coordination ability.
Even when LLMs are given communication channels, they may not use them effectively for long-horizon planning. In alem, ablations showed that communication was the largest contributor to coordination, but memory and reasoning only helped when used to maintain multi-step plans [2]. This implies that without explicit mechanisms for maintaining shared plans over time, LLMs lose track of coordination, a failure that short-horizon tasks simply don't reveal.
What would a better coordination benchmark look like?
To reveal true long-horizon coordination ability, benchmarks must combine open-endedness, long horizons, and explicit coordination demands. The alem benchmark does this by embedding procedurally generated coordination tasks, soft specialisation, communication, and controllable difficulty into a long-horizon survival world [2]. Its results—showing that even frontier models fail—demonstrate that such benchmarks are necessary to identify coordination as a distinct bottleneck, separate from single-agent capabilities.
Additionally, benchmarks should separate coordination-specific rewards from task rewards, as alem does, to isolate coordination failures from general task failures [2]. This is crucial because, as the 2023 benchmark showed, LLMs can be robust to unseen partners in zero-shot coordination, but that robustness may not extend to long-horizon settings where plans must be updated dynamically [5]. Without such distinctions, a model that is individually competent but coordination-poor will look better than it is.
About These Sources
This answer is built on 5 studies (2 peer-reviewed, 3 preprints) — published from 2023 to 2026, 3 from 2024 or later, 1 in Q1 journals, collectively cited 57 times — selected as the most relevant from 7 studies that passed quality screening, drawn from 26 papers retrieved from a database of over 500 million.
Sources used in this answer
SyncPlan: Long-Horizon LLM Coordination with Explicit Synchronization and Adaptive Correction
SyncPlan, a plan-execute-correct framework, achieves state-of-the-art success rates on Overcooked and Honor of Kings while using less than 0.05% of the wall-clock runtime of existing LLM coordinators, showing that explicit synchronization and adaptive correction can mitigate long-horizon coordination failures.
Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
In the alem benchmark, 13 modern LLMs averaged only ~6% normalized return in open-ended long-horizon coordination, with GPT-5.4-High showing strong base-task reward but much lower coordination reward, and ablations identifying communication as the largest contributor to coordination.
Open-ended coordination for multi-agent systems using modular open policies
A reinforcement learning approach using policy blending with goal inference outperformed a joint-reward baseline in open environments with changing agents and tasks, highlighting the importance of adaptive goal weighting for coordination in open-ended settings.
Long-range inhibition synchronizes and updates prefrontal task activity
In mice, selectively inhibiting callosal projections of parvalbumin-expressing neurons impaired rule-shift learning and desynchronized gamma-frequency activity, revealing a neural mechanism for switching between maintaining and updating task representations—an analogy to how coordination may require flexible updating of plans.
LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models
The LLM-Coordination Benchmark found that LLM agents excel in coordination games relying on environmental variables but struggle with Theory of Mind reasoning and joint planning, and are robust to unseen partners in zero-shot coordination.