What can competing-agent management tasks fail to reveal about long-horizon management with competing agents?

What do competing-agent benchmarks actually measure?

Competing-agent management tasks, like the FM-Bench football management simulation, are designed to test long-horizon decision-making by pitting agents against each other in a shared, evolving world. They measure observable behaviors: how well an agent manages resources, negotiates contracts, and adapts to changing conditions. In FM-Bench, agents run a football club for 20 in-game years, making hundreds of decisions, and the final score is determined by a deterministic engine—no human judge or LLM evaluator [2]. This setup is valuable because it provides a clear, objective outcome and reveals behavioral differences: higher-scoring agents reduced slow-payoff investments near the end, kept cash invested rather than idle, and opened contract renewals early [2]. However, these benchmarks primarily capture what an agent does, not why it succeeds or fails. They show that some agents are better at long-horizon management, but they don't explain the underlying cognitive or algorithmic reasons for the differences.

The limitation is that these tasks are often too structured and well-defined to expose the full range of challenges in real-world long-horizon management. For instance, in FM-Bench, all 15 models completed the entire 20-year horizon, while scripted baselines died out—suggesting that the task is solvable with enough compute, but the benchmark doesn't reveal whether the agents are truly planning ahead or just reacting to immediate feedback [2]. This is a key gap: the benchmark measures outcomes, but not the planning process. A study on planning-centric analysis of LLM agents found that step-by-step reasoning often leads to myopic decisions that are hard to reverse, a failure mode that would be invisible in a benchmark that only scores final outcomes [4].

What hidden failure modes do these benchmarks miss?

Competing-agent tasks often fail to reveal deep-seated issues like memory mismanagement and the inability to learn from experience. In FM-Bench, researchers found that no model learned the market's hidden prices from hundreds of rejected bids, and self-managed memory failed in two opposite ways: an archive that only grew or a plan rewritten every season [2]. These are critical failures for long-horizon management, but they are not directly scored by the benchmark—they only manifest as suboptimal decisions over time. Similarly, a study on multi-agent LLM systems identified 14 distinct failure modes, including inter-agent misalignment and task verification issues, which are often not captured by simple outcome-based benchmarks [5]. These failures are systemic and require sophisticated solutions, but they are invisible if you only look at the final score.

Another hidden issue is the inability to handle delayed consequences. A planning-centric analysis showed that LLM agents often make locally optimal choices that lead to early commitments that are difficult to recover from, a problem that compounds over long horizons [4]. This is not something a competing-agent benchmark would directly measure, because it doesn't track the decision-making process—it only sees the final outcome. The FM-Bench study also noted that the order of model performance settled only late in the horizon, suggesting that early decisions have cumulative effects that are hard to predict [2]. This means that even if an agent performs well on a benchmark, it might be doing so through luck or short-term optimization rather than genuine long-horizon planning.

Why outcome scores alone are insufficient

Outcome scores, like the final score in FM-Bench or pass rates in terminal benchmarks, provide a single number that summarizes performance, but they obscure the nuances of long-horizon management. For example, Long-Horizon-Terminal-Bench found that even the strongest model achieved only 15.2% partial success (at a threshold of 0.95) and 10.9% perfect success, while the average across models was 4.3% and 1.7% respectively [1]. These numbers are stark, but they don't tell you why agents fail—whether it's due to poor planning, memory issues, or inability to handle long contexts. The benchmark's design with dense intermediate rewards helps, but it still focuses on task completion rather than the underlying cognitive processes.

To truly understand long-horizon management, we need benchmarks that go beyond final outcomes and measure intermediate progress, decision quality, and adaptability. The Emergence World platform attempts this by running continuous multi-agent simulations over weeks, with agents grounded in live data and governed by democratic mechanisms [3]. In a 15-day study, identical roles and starting conditions produced radically different outcomes, from stable governance to total population collapse, highlighting that long-horizon dynamics like behavioral drift and cross-influence only emerge over time [3]. This suggests that competing-agent tasks, which typically run for hours or days, may miss the slow-burning failures that matter in real-world deployments. The FM-Bench study also noted that token spend predicted nothing about performance, indicating that raw compute is not a proxy for managerial skill [2].

About These Sources

This answer is built on 5 studies (all preprints) — published from 2025 to 2026, 5 from 2024 or later — selected as the most relevant from 7 studies that passed quality screening, drawn from 29 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions