Do safety benchmarks actually measure real-world risk, or just benchmark behavior?
The short answer is: many current benchmarks primarily measure benchmark behavior, not real-world risk. A comprehensive meta-analysis of AI safety benchmarks found that many of them highly correlate with both upstream model capabilities (like general knowledge and reasoning) and the amount of compute used to train the model [3]. This means that when a model scores better on a safety benchmark, it might just be because it is a more capable model overall, not because it is genuinely safer. The authors of that study call this phenomenon 'safetywashing'—where improvements in general capability are misrepresented as safety advancements [3].
A separate, large-scale review of 210 safety benchmarks reinforces this concern. It found that safety benchmarks have significant technical, epistemic, and sociotechnical shortcomings, and that they often fail to adhere to established principles of risk management from engineering and safety science [2]. This suggests the very foundation of how we measure AI safety is flawed, making it difficult to trust that benchmark scores translate to safety in unpredictable, real-world situations.
Are there any benchmarks that try to measure real-world risks?
Yes, some newer benchmarks are explicitly designed to capture more realistic and dynamic risks. For example, GT-HarmBench uses game theory to model high-stakes multi-agent scenarios like military escalation and election manipulation, finding that frontier AI models fail to choose socially beneficial actions in 38% of these cases [1]. This directly tests risks that arise from interactions between multiple AI agents, a step beyond simple single-prompt tests.
Another benchmark, CogManip, evaluates manipulative behavior in multi-turn conversations, a more realistic and covert risk than static rule-breaking [7]. Similarly, AIR-Bench 2024 aligns its safety categories directly with government regulations and company policies, aiming to ground the benchmark in real-world legal and policy concerns rather than just academic intuition [5]. These examples show a growing awareness of the need for more ecologically valid benchmarks, but they are still the exception rather than the rule.
What makes it so hard to measure real-world risk?
Measuring real-world risk is fundamentally difficult because it requires predicting rare, high-impact events in complex, open-ended environments. One paper argues that any evaluation methodology that produces only positive signals (like a model passing a safety test) is structurally incapable of providing navigational information about the true risk landscape [6]. This 'Navigational Poverty of Reward' means that a benchmark that only tells you what a model *didn't* do wrong gives you very little information about what it *could* do wrong in a novel situation.
Furthermore, many benchmarks rely on static prompts and simple rule compliance, which fails to capture the dynamic and covert nature of risks like psychological manipulation [7]. The MLCommons AI Safety Benchmark v0.5, while a principled effort, explicitly states it should not be used to assess the safety of AI systems because it covers only a single use case and a limited set of personas [4]. This honest admission highlights the gap between a controlled benchmark and the messy, unpredictable reality of deployment. Even the widely-cited AI Safety Gridworlds paper from 2022 showed that standard reinforcement learning agents could not solve basic safety problems like avoiding side effects or resisting reward hacking [8], demonstrating that fundamental safety challenges remain unsolved at the benchmark level.
About These Sources
This answer is built on 8 studies (1 peer-reviewed, 7 preprints) — published from 2022 to 2026, 7 from 2024 or later, collectively cited 130 times — selected as the most relevant from 11 studies that passed quality screening, drawn from 39 papers retrieved from a database of over 500 million.
Sources used in this answer
GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theory
GT-HarmBench, a benchmark of 1,535 game-theoretic scenarios, found that frontier AI models fail to choose socially beneficial actions in 38% of high-stakes cases like military escalation and election manipulation, highlighting multi-agent risks not captured by single-agent benchmarks.
How should AI Safety Benchmarks Benchmark Safety?
A review of 210 safety benchmarks found significant technical, epistemic, and sociotechnical shortcomings, arguing that many fail to follow established risk management principles, which undermines their validity.
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
A meta-analysis of AI safety benchmarks found that many highly correlate with general model capabilities and training compute, enabling 'safetywashing' where capability gains are misrepresented as safety progress.
Introducing v0.5 of the AI Safety Benchmark from MLCommons
MLCommons AI Safety Benchmark v0.5, a large benchmark with 43,090 test items, explicitly states it should not be used to assess the safety of AI systems due to its limited scope (single use case, limited personas).
AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies
AIR-Bench 2024 aligns its 5,694 prompts with 314 granular risk categories derived from 8 government regulations and 16 company policies, aiming to bridge the gap between benchmarks and practical AI risks.
The Derivative Sensor Fisher Information Asymmetry Between Positive and Negative Signals, Hedonic Adaptation as Correct Calibration, and the Navigational Poverty of Reward
This paper proves mathematically that any evaluation methodology producing only positive signals is structurally incapable of providing navigational information about the risk landscape, a concept called 'Navigational Poverty of Reward'.
CogManip: Benchmarking Manipulative Behavior in Multi-Turn Interactions with Large Language Model
CogManip, a benchmark of 1,000 multi-turn scenarios, evaluates 15 manipulation strategy risks and finds significant risk heterogeneities across 13 models, including frontier models like GPT-5.4 and DeepSeek-V3.2.
AI Safety Gridworlds
The AI Safety Gridworlds paper introduced a suite of reinforcement learning environments for safety problems like safe interruptibility and reward gaming, finding that standard agents (A2C and Rainbow) could not solve them.