Why AI races make the world less safe—and why cooperation is so hard
The central trade-off in AI governance is between racing ahead for competitive advantage and slowing down to manage risks. Simulation games like 'Intelligence Rising'—run 43 times over four years—show that when states (e.g., U.S., China) or firms (e.g., OpenAI, Google) compete, the result is almost always destabilizing: increased risk of safety failures or geopolitical conflict, and a dramatic drop in the likelihood of positive futures [1]. The games also reveal why cooperation is so rare: actors have strong default incentives to compete, and those with the resources to understand the risks are few, giving them outsized power to either worsen or improve outcomes [1].
This finding is not just theoretical. It aligns with the broader regulatory literature: [4] argues that frontier AI models—those with potentially dangerous capabilities—pose a distinct challenge because dangerous capabilities can emerge unexpectedly and are hard to control once deployed. Both [1] and [4] converge on the same conclusion: without deliberate coordination, the default path is risky. The simulation evidence adds a behavioral layer—showing that even when participants understand the risks, they still race, which is why policy cannot rely on goodwill alone [1].
The coming flood of models will break today's compute-based rules
Current regulations, like the EU AI Act and the U.S. AI Diffusion Framework, use absolute training compute thresholds to decide which models face extra requirements—for example, 1025 FLOP (floating-point operations) in the EU and 1026 FLOP in the U.S. [2]. But a forecast to 2028 predicts that the number of models exceeding these thresholds will grow superlinearly—each year will see more new models captured than the year before—leading to an estimated 103–306 models above the EU threshold and 45–148 above the U.S. threshold by the end of 2028 [2]. That's a huge number for regulators to track, and it suggests that absolute thresholds will quickly become unwieldy.
The same forecast offers a more stable alternative: using a relative threshold—for example, capturing all models within one order of magnitude of the largest training run to date—would bring in a consistent 14–16 models per year from 2025 to 2028 [2]. This is a practical insight for policymakers: instead of chasing a moving target, they could anchor rules to the frontier itself. It also means that over the next two years, we should expect debates about how to define 'frontier' models, because the current definitions will soon be overwhelmed by sheer numbers.
What governance will actually look like in the next two years
The simulation evidence and regulatory analyses point to a shift from voluntary self-regulation to mandatory oversight. [4] proposes three building blocks: standard-setting, registration and reporting, and compliance mechanisms—including licensing for frontier models. The simulations reinforce this by showing that international agreements, while possible, are fragile and often fail without enforcement [1]. So, expect to see governments moving from asking companies to 'be safe' to requiring them to register models, conduct pre-deployment risk assessments, and submit to external scrutiny [4].
But there's a catch: the governance landscape is changing underneath these plans. [3] warns that open-source and local AI models—running on personal devices—are now only months behind proprietary models, and they are invisible to regulators and stripped of safety constraints. This undermines the assumption that frontier AI will always live in data centers. Over the next two years, policy will need to adapt to a decentralized ecosystem, using tools like content provenance tracking and polycentric governance—where multiple local and international bodies share oversight—rather than relying solely on top-down rules [3].
International coordination is another key theme. [5] draws lessons from the IAEA's nuclear safety regime, arguing for standardized global norms to reduce discrepancies between national regulations. However, it also acknowledges that AI differs from nuclear technology, so a direct copy won't work; we need a flexible, continuously updated framework [5]. The simulations agree: positive futures almost always require coordination between actors who have strong incentives to compete [1]. So, over the next two years, look for efforts to build international consensus—perhaps through bodies like the UN or new AI-specific agencies—even if they start with non-binding agreements.
About These Sources
This answer is built on 5 studies (4 peer-reviewed, 1 preprint) — published from 2023 to 2025, 4 from 2024 or later, 2 in Q1 journals, collectively cited 103 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 46 papers retrieved from a database of over 500 million.
Sources used in this answer
Strategic insights from simulation gaming of AI race dynamics
In 43 runs of the 'Intelligence Rising' simulation over four years, AI races between states and firms consistently increased risks of safety or geopolitical failure, while positive outcomes almost always required international cooperation that was difficult to achieve.
Trends in Frontier AI Model Count: A Forecast to 2028
A forecast to 2028 estimates that 103–306 models will exceed the EU's 1025 FLOP threshold and 45–148 will exceed the U.S.'s 1026 FLOP threshold, with the count growing superlinearly each year; a relative threshold would capture a stable 14–16 models annually.
Local AI Governance: Addressing Model Safety and Policy Challenges Posed by Decentralized AI
Local, open-source AI models now lag only months behind proprietary ones and can run on personal devices, making them invisible to regulators and stripping safety constraints; the paper proposes technical safeguards and polycentric governance to manage this shift.
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Frontier AI models pose distinct regulatory challenges—unexpected dangerous capabilities, difficulty preventing misuse, and rapid proliferation—and require standard-setting, registration/reporting, and compliance mechanisms, including licensing and pre-deployment risk assessments.
Towards an international regulatory framework for AI safety: lessons from the IAEA’s nuclear safety regulations
Drawing on IAEA nuclear safety regulations, the paper argues for standardized international AI safety norms to reduce regulatory discrepancies, while noting that AI's unique risks require a tailored, flexible framework rather than a direct copy.