Do frontier AI evaluations actually measure real-world harm, or just test scores?
Most current evaluations of frontier AI systems are designed to test specific capabilities—like answering questions, generating text, or solving math problems—on standardized benchmarks. These benchmarks are useful for comparing model performance, but they do not capture how a model might cause harm in the messy, unpredictable real world. A 2024 paper on internal audit functions points out that dangerous capabilities can arise unexpectedly and go undetected during standard testing, meaning a model that passes all benchmarks could still cause serious harm once deployed [2]. This is a fundamental limitation: benchmarks measure what we think to test, not what the model might actually do when misused or when it interacts with complex systems.
Another paper on responsible reporting for frontier AI development argues that developers have significant access to safety-critical information about their systems, but that information is rarely shared with regulators or the public [3]. Without that visibility, external evaluations can only assess what developers choose to reveal, leaving real-world risks—like a model being used to generate disinformation or automate cyberattacks—largely invisible. The paper calls for mandatory reporting of safety-critical information to close this gap [3].
A 2023 paper on deepfakes and disinformation highlights that generative AI can craft convincingly fabricated content that evades detection, and that current benchmarks for detecting such content are quickly outdated as models improve [4]. This means a model that scores well on a deepfake detection benchmark today might still be used to create undetectable disinformation tomorrow. The paper advocates for continuous, adaptive defense mechanisms rather than one-time benchmark evaluations [4].
Why even a large increase in real-world harm might go unnoticed by current evaluations
A key finding from a 2026 study on cybercrime damages reveals a stark problem: the baseline data for real-world harm is so noisy that even a significant AI-driven increase would be invisible. The study estimates total global cybercrime damages at roughly $500 billion per year, but with a 90% confidence interval spanning from $100 billion to $1 trillion [1]. This means the uncertainty is enormous—the true number could be ten times higher or lower than the estimate. The authors calculate that an AI-driven increase of about 20% would add $100 billion or more in damages, but they conclude that 'cybercrime data remains too incomplete for such incremental increases to be directly detectable' [1]. In other words, even if frontier AI systems caused a massive surge in cybercrime, current evaluation methods would not be able to measure it.
This finding directly answers the question: frontier AI evaluations do not measure real-world risk because the real-world data needed to detect changes is too imprecise. The paper recommends using victimization surveys—which ask people directly about losses—rather than relying on law enforcement reports or macroeconomic models, but even those surveys have wide error margins [1]. So while benchmarks can tell us if a model is getting 'better' at certain tasks, they cannot tell us whether that improvement translates into more harm in the real world.
What would it take to make evaluations measure real-world risk?
Several papers converge on the idea that closing the gap between benchmarks and real-world risk requires stronger governance, not just better tests. A 2023 paper on frontier AI regulation proposes three building blocks: standard-setting processes to define what safety means, registration and reporting requirements to give regulators visibility into development, and compliance mechanisms to enforce safety standards [5]. The paper specifically recommends pre-deployment risk assessments, external scrutiny of model behavior, and post-deployment monitoring [5]. These are all steps that go beyond benchmark evaluations and attempt to measure how a model might actually behave in the wild.
The internal audit paper adds that frontier AI developers need an independent internal audit function to evaluate the adequacy of their risk management practices [2]. This is not about testing the model itself, but about testing the processes around it—whether the company is actually looking for real-world risks, whether it has controls in place, and whether the board of directors has an accurate understanding of the current level of risk [2]. The paper notes that internal audit can identify ineffective risk management practices and serve as a contact point for whistleblowers, but it also warns that audit can be captured by senior management and that its benefits depend on the ability of individuals to identify ineffective practices [2].
Taken together, these papers suggest that real-world risk measurement is not just a technical problem of better benchmarks—it is a governance problem. Without mandatory reporting, independent oversight, and continuous monitoring, even the most sophisticated evaluations will only measure what we already know to test, not the unpredictable ways frontier AI could cause harm.
About These Sources
This answer is built on 5 studies (4 peer-reviewed, 1 preprint) — published from 2023 to 2026, 3 from 2024 or later, 1 in Q1 journals, collectively cited 159 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 49 papers retrieved from a database of over 500 million.
Sources used in this answer
Global Cybercrime Damages: A Baseline for Frontier AI Risk Assessment
Estimates global cybercrime damages at ~$500 billion/year (90% CI: $100B–$1T) and finds that even a 20% AI-driven increase would be undetectable with current data, showing that real-world harm baselines are too imprecise for current evaluations to measure incremental risk.
Frontier AI developers need an internal audit function
Argues that dangerous capabilities in frontier AI can arise unpredictably and undetected, and that current risk governance is inadequate; proposes an internal audit function to evaluate risk management practices, not just model benchmarks.
Responsible Reporting for Frontier AI Development
Recommends mandatory reporting of safety-critical information by frontier AI developers to governments and civil society, as current evaluations lack visibility into real-world risks that developers alone can see.
Deepfakes, Misinformation, and Disinformation in the Era of Frontier AI, Generative AI, and Large AI Models
Reviews how generative AI can create undetectable deepfakes and disinformation, and argues that current detection benchmarks are quickly outdated, requiring continuous adaptive defenses rather than one-time evaluations.
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Proposes regulatory building blocks—standard-setting, registration/reporting, and compliance enforcement—to manage frontier AI risks, emphasizing pre-deployment risk assessments and post-deployment monitoring beyond benchmark tests.