Can AI Agents Learn Heuristic Strategies to Top a Game Competition?

A competition ladder instead of a single match

The paper asks whether an AI agent can act as a learning engine for adversarial games: reading rules, choosing opponents, studying replays, and revising the code that plays the game until it climbs a leaderboard. This is the authors' Adversarial Heuristic Learning (AHL) paradigm, an extension of heuristic learning in which experience is stored as edits to a program rather than as changes to model parameters.

AAArena makes that question measurable. It contains 12 authentic adversarial games and 1,920 archived human programs, and its protocol is modeled on real-world game competitions. The agent's goal is its highest ranking within fixed match and evaluation budgets, so success is defined by ladder position against frozen human programs, not by win rate in a single match.

Two feedback channels drive each revision

The agent works through two interfaces. Small matches let it pick specific opponents and inspect detailed replays, which the paper describes as targeted diagnosis of strategic hypotheses. Full-pool evaluations run the current program against the complete opponent pool and return Elo, rank, and per-opponent outcomes, giving a common reference for comparing snapshots.

Feedback differs by channel: small matches return dense feedback, while full-pool evaluations return binary feedback. Both evaluate immutable programs through the same game referee, and the agent's revisions are checked by a validator for legality and runtime before they can be used. The agent may not search the internet or read rivals' code, so improvement has to come from rules, replays, and match results.

Six of twelve ladders were topped

Across 7 evaluated model and harness configurations, Opus5.5 with Claude Code earned 6 gold medals, meaning rank-1 finishes on 6 ladders. No evaluated configuration topped the remaining 6 human ladders. That split is the paper's headline result: heuristic learning is enough to reach the top in half of these games, and insufficient in the other half under the same protocol.

The comparison is between configurations and ladders, not a claim that any model solved the games in general. A gold medal reflects the retained champion's rank against a frozen pool of human programs within the allotted budgets, so it measures competitive standing in this benchmark rather than play strength against arbitrary opponents.

Complex rule specifications predict weaker results

The paper reports that performance is generally weaker in games with more complex rule specifications. It measures that complexity two ways: abstract syntax tree (AST) node count for the rule description, and rule atoms (RA), which count independently changeable rule propositions. Table 1 orders the 12 games by increasing AST node count, from SnakeGo at 141 nodes and 1,538 rule atoms up to AntWar2 at 6,368 nodes and 1,008 rule atoms.

The authors are explicit that both measures describe the size of the rule description, not the state-space size or the difficulty of optimal play. So the pattern links weaker rankings to larger rule descriptions, and the paper's qualitative analysis points to game understanding, strategy implementation, and long-horizon policy development as the persistent challenges behind it.

Opponent selection and dense feedback help

Further experiments isolate what supports policy improvement. The paper reports that opponent selection and dense feedback both support improvement, which fits the two-channel design: choosing whom to test against concentrates evidence on a specific hypothesis, and dense small-match feedback says more per interaction than a binary full-pool outcome.

Replay source matters less than one might expect. Agents improved by learning from both on-policy replays of their own matches and off-policy replays of other players' matches. That result widens the usable evidence base beyond the agent's own games, though the paper presents it as an ablation finding within AAArena rather than a general law about imitation.

Where these rankings do and do not transfer

The evidence covers 12 specific games, a frozen pool of archived human programs, and fixed budgets for small matches and full-pool evaluations. Pool sizes count executable programs, not unique human participants, because some entrants submitted several related versions. Rankings therefore compare against programs, and a ladder position should not be read as a standing against a fresh field of human players.

The paper also notes that its complexity measures depend on the chosen rule language. A game that looks simpler under AST or RA counting is not necessarily easier to play well, so the weaker results on complex specifications are a pattern within this suite rather than a calibrated difficulty scale.

Judge agent entries by revisions, not just medals

The most useful distinction in this paper is between the two halves of the ladder. Six gold medals show that an agent can convert rules and replays into a competitive program when the rule specification is tractable; six untaken ladders show that the same loop stalls when the agent must understand a larger rule set, implement a coherent strategy, and sustain it over a long horizon.

For anyone evaluating an agent entry in a competition like this, the paper's evidence favors looking at the revision trajectory and the game's rule complexity alongside the final rank. A rank-1 finish on a compact rule set and a mid-ladder finish on a large one describe different capabilities, and the paper's ablations suggest the levers that separate them are opponent selection and dense feedback rather than more matches alone.

Sources used in this answer