Should benchmark designers train on the test set to expose non-visual shortcuts?

Blind tests show priors, not learnable patterns

A multimodal model can answer many vision-centric benchmark questions without the image, relying on linguistic priors and skewed answer distributions. The standard safeguard is the blind test: run the benchmark with vision disabled and see what the model already answers. That measures pretrained knowledge, but it does not measure what a model could pick up from the test set's own regularities, such as question templates, repeated object pairings, or answer-option structure.

The paper's proposal is to close that gap by fitting a probe to the test set's non-visual input and measuring what it learns. The authors frame this as "training on the test set," but explicitly address it to benchmark designers as a held-out, pre-release audit of the benchmark artifact, not a recipe for training deployed models on test answers. The probe audits the data, not a model.

How the Test-set Stress-Test scores a benchmark

The Test-set Stress-Test (TsT) applies k-fold cross-validation to the test set's non-visual input, using k=5 and seed 42 with one repeat. Each question is scored only in its held-out fold, so the probe never sees the answer it is being evaluated on. The result is a benchmark-level TsT score plus a per-question bias score s(x).

Two diagnostics implement this. TsT-LLM LoRA-tunes Qwen2-7B-Instruct on question and answer-option text, with rank 8, alpha 16, one epoch per fold, and a learning rate of 2×10⁻⁴. TsT-RF is a random forest on hand-crafted, benchmark-specific features such as object categories, answer format, option count, and training-fold frequency statistics; it adds a fast, interpretable audit. Features keyed on the held-out question's gold answer are excluded to avoid label leakage.

Template-based benchmarks leak the most

On VSI-Bench and CV-Bench, both built from question templates, the held-out TsT-LLM score is 17.9 and 13.1 points above the same blind Qwen2-7B's zero-shot score. That gap is the paper's central evidence: a text-only model trained on the test set's own questions and options learns regularities that a zero-shot blind model does not exploit.

The CV-Bench number is deliberately conservative. Many zero-shot CV-Bench answers begin with a word rather than an option letter and so receive chance credit; scoring those as wrong would give a ΔTsT of about +18.4 instead of +13.1. The authors report the smaller value because the larger one would credit TsT for learning the answer format. Reruns also vary: VSI-Bench scores land at 42.6–42.8 (ΔTsT≈18), and the CV-Bench gain ranges from about 13 to 18 points across reruns and fold assignments.

MMMU blind success is knowledge, not a learnable shortcut

On knowledge-intensive MMMU the probe learns little, even though GPT-4o correctly answers 52.3% of its multiple-choice questions without the image. The authors read this contrast as evidence that blind success on MMMU reflects pretrained knowledge rather than learnable test-set patterns. A high blind score and a high TsT score therefore answer different questions: the first says what a model already knows, the second says what the released artifact makes learnable.

This also means a small ΔTsT is only a lower bound on learnable non-visual signal. A stronger diagnostic might find more, and no partial-input probe detects artifacts that depend on the omitted image. Pruning MMMU by Δs(x) behaves like random removal: dropping 100 of 900 questions moves the gap from 6.7 to 7.5, within the random-removal range of 4.62 to 8.75 across 1,000 seeds.

Pruning the most biased questions widens the gap

Iterative Bias Pruning (IBP) repeatedly removes the questions with the highest s(x) and re-diagnoses the rest. On VSI-Bench, the designer-in-the-loop pilot removes 54.0% of questions (2,768 of 5,130, leaving 2,362) and widens a fine-tuned LLaVA-Video-7B's vision-blind gap from 12.4 to 16.7, against 12.4±0.6 for random removal at the same rate.

The widening is not uniform across models. It is largest for Cambrian-S (19.2 to 26.0) and the fine-tuned LLaVA-Video-7B, the two with the highest original blind scores, and it also appears for InternVL3-9B (11.1 to 13.2), whose language model is not from the Qwen family. For models with the lowest original blind scores the gap barely moves, and for LLaVA-OneVision-7B it slightly narrows (9.0 to 8.5).

Automated IBP is weaker than the hand-tuned pilot. Per-type IBP at the same 54% removal rate reaches gaps of 14.1 to 14.7, and per-format IBP at a budget of 1,000 questions removes 19.5% and leaves the gap at 12.4, close to random removal at that rate.

The released debiased set trades coverage for a wider gap

VSI-Bench-Debiased lowers a fine-tuned LLaVA-Video-7B's blind score from 44.7 to 32.0 while its vision score falls only from 57.1 to 48.7, so the gap grows from 12.4 to 16.7. Fine-tuning on VSI-Train-10k raises vision by 17.4 points and blind by 11.7 on the debiased set, versus 20.4 and 18.8 on the original, meaning more of the gain there requires vision.

The cost is a changed question mix. The pilot uses hand-written per-type filters and hand-set budgets, removing 85.7% of medium relative-direction questions but only 2.3% of easy ones, and keeps all ten question types. The authors present this as a limitation of the pilot that motivates an automated successor, and they release the debiased set separately rather than as a drop-in replacement.

Where the diagnosis stops

Folds split questions, not the scenes or images behind them, so random folds also credit regularities shared by questions about one scene. Grouping VSI-Bench folds by scene lowers TsT-RF from 43.0 to 38.8, mostly in appearance order, while the TsT-LLM gain changes little (17.9 to 17.0 points). The authors recommend reporting both random- and grouped-fold scores.

The two diagnostics also disagree about which questions to remove: their removed sets have a Jaccard overlap of 0.060, and swapping Qwen2-7B for LLaMA-3-8B in TsT-LLM gives Δs(x) rankings with Spearman 0.58. TsT-RF needs hand-crafted features per benchmark, and the flagged questions have not been systematically reviewed. VSI-Bench-Debiased reduces non-visual shortcuts without removing them, and pruning can create new answer priors; the pruned set was not re-run through TsT.

What designers should report before release

The paper's practical priority is a pre-release, held-out audit of the benchmark artifact, reported alongside the benchmark itself. The most decision-relevant comparison is between benchmarks, not between models: template-based VSI-Bench and CV-Bench show large learnable gains (17.9 and 13.1 points), while MMMU shows little despite GPT-4o answering 52.3% of its multiple-choice questions blind. A benchmark with a high blind score but a low TsT score is measuring knowledge; one with a large TsT gain is handing models a shortcut.

For designers deciding whether to prune, the paper's own numbers set the bar. The hand-tuned pilot widened the gap to 16.7, automated per-type IBP reached 14.1 to 14.7, and per-format IBP at 19.5% removal left it at 12.4, essentially unchanged from the original 12.4. Pruning that does not beat random removal at the same rate has not demonstrated a benefit, and any refined benchmark should be re-diagnosed with grouped folds before release.

Sources used in this answer