When providers hide data, benchmarks become easier to game — but they were already vulnerable
The less you know about a model's training data, the harder it is to tell whether a benchmark score reflects genuine safety or just clever overfitting. A major interdisciplinary review of about 110 studies on benchmarking practices found that data contamination — where a model has already seen benchmark test examples during training — is a widespread problem that undermines trust in scores [3]. When model providers disclose less data, you cannot check for this contamination, making it easy for a provider to claim high safety scores that don't transfer to real-world use. The same review also highlights that benchmarks are often gamed by developers who optimize for the test rather than for actual safety, a problem that worsens when the evaluation is a black box [3].
However, even with full data transparency, benchmarks have deep structural flaws. A separate review of 210 safety benchmarks found that many fail to connect their measurements to real-world risk, because they ignore established risk-management principles and don't map what can and cannot be measured [1]. This means that even if you know everything about the training data, the benchmark itself might be measuring the wrong thing. The two reviews together suggest that less disclosure amplifies an existing weakness, but it doesn't create the problem from scratch.
Benchmarks miss regional and cultural safety issues — and less data hides those gaps
Safety benchmarks are not universal; they often reflect Western cultural assumptions, and when model providers disclose less data, you cannot tell whether the model has been tuned to avoid region-specific harms. A study that created a culturally adapted bias benchmark for Pakistan (PakBBQ) found that multilingual LLMs showed 12% higher accuracy on average when given explicit disambiguation cues, and that the models exhibited stronger counter-bias behavior in Urdu than in English [2]. This means that a benchmark designed for one culture can miss serious safety issues in another, and without knowing what data the model was trained on, you cannot assess whether those gaps have been addressed.
The same study also found that simple changes in question framing — asking questions negatively versus positively — reduced stereotypical responses, showing that benchmark results are highly sensitive to prompt wording [2]. When model providers disclose less data, you lose the ability to understand whether the model's performance on a safety benchmark is robust across different cultural contexts or just a fluke of how the test was phrased. This is a concrete example of the broader problem identified in the 210-benchmark review: benchmarks often fail to connect their objectives to the real world, and less data makes that disconnect harder to detect [1].
Even the tools used to evaluate safety can be biased — and less data hides that too
Safety benchmarks often rely on automated classifiers to judge model outputs, but those classifiers can have their own biases that distort results. A study examining open-ended generation bias benchmarks (like BOLD and SAGED) found that the classifiers used to analyze LLM outputs treated demographic descriptors unequally, leading to unfair conclusions about which models are biased [4]. The researchers used counterfactual testing — changing stereotype-related prefixes in prompts — and found that the classifiers' predictions varied significantly across demographic groups, meaning the benchmark was measuring the classifier's bias as much as the model's [4].
When model providers disclose less data, you cannot audit whether these hidden biases in the evaluation tools are inflating or deflating safety scores. The study explicitly calls for more robust bias metric models, but if the provider doesn't share details about the evaluation pipeline, the problem is invisible [4]. This aligns with the broader finding from the 110-study review that benchmarks often fail to distinguish signal from noise, and that inadequate documentation is a systemic flaw [3]. Less data disclosure means less documentation, which means more noise and less trust in any safety claim.
About These Sources
This answer is built on 4 studies (1 peer-reviewed, 3 preprints) — published from 2024 to 2026, 4 from 2024 or later — selected as the most relevant from 4 studies that passed quality screening, drawn from 32 papers retrieved from a database of over 500 million.
Sources used in this answer
How should AI Safety Benchmarks Benchmark Safety?
A review of 210 safety benchmarks found that many fail to follow established risk-management principles and do not map what can and cannot be measured, undermining their validity even with full data disclosure.
PakBBQ: A Culturally Adapted Bias Benchmark for QA
A culturally adapted bias benchmark for Pakistan (PakBBQ) showed that LLMs had 12% higher accuracy with disambiguation and stronger counter-bias behavior in Urdu than English, revealing that benchmarks miss regional safety issues and are sensitive to prompt wording.
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
An interdisciplinary meta-review of about 110 studies identified systemic flaws in benchmarking, including data contamination, gaming of results, and inadequate documentation, which worsen when model providers disclose less data.
Assessing Bias in Metric Models for LLM Open-Ended Generation Bias Benchmarks
A study of open-ended generation bias benchmarks found that the classifiers used to evaluate LLM outputs themselves treat demographic descriptors unequally, skewing results regardless of how much the model provider reveals.