A benchmark that strips reasoning away
Most visual benchmarks for multimodal large language models (MLLMs) mix perception with reasoning and domain knowledge in fields such as engineering, medicine, and art. That makes it hard to tell whether a model actually sees an image or is filling gaps with language priors and learned facts. Percept-V isolates the first step: 6,000 program-generated images across 30 domains, built from simple geometric objects such as circles, triangles, and squares arranged in grids, rows, and mazes.
The tasks require minimal reasoning and no specialized knowledge. Ground-truth answers are computed during image generation, so there is no annotation ambiguity, and programmatic generation also removes the risk of test-set contamination. Each domain contains 200 images whose problem size varies through object count, grid size, or the number of processing steps, which lets the authors measure accuracy as visual complexity rises.
Seven perception skills borrowed from cognitive science
The dataset is organized around TVPS-4, a cognitive-science framework that divides human visual perception into seven skills, including visual discrimination, visual memory, visual sequential memory, visual figure ground, visual form constancy, visual closure, and visual spatial relationship. Each Percept-V domain tests one to three of these skills, and a single domain can appear under several skills because solving it may demand more than one perceptual operation.
This mapping matters for reading the results. A score is not a generic "vision score"; it is an accuracy on a specific skill, aggregated over the domains assigned to it. For example, visual spatial relationship covers tasks such as counting circles above and below a rectangle or solving a maze, while visual closure covers matching outlines and shadows. The paper reports both per-skill averages and per-domain accuracies, and these two levels should not be treated as interchangeable.
Top models average 41% across the full dataset
The authors evaluated four proprietary models (GPT-5-mini, GPT-4o, o4-mini, Gemini 2.5 Flash) and four open-source models (Qwen 2.5 VL Instruct-7B, DeepSeek VL2 Tiny, Qwen-3VL-8B-Thinking, Gemma-4-31B-it), plus a human study. In the one-shot setting, full-dataset accuracy was 52.13% for GPT-5-mini, 51.01% for o4-mini, 43.79% for Gemini 2.5 Flash, and 18.21% for GPT-4o, giving a four-model average of 41.28%.
The paper's discussion states that the best average performance of any model was 66.9%, and that human performance was very high by comparison. The gap is the central finding: models that handle complex reasoning tasks still miss simple perceptual questions. The authors' prior expectation was the opposite, that modern MLLMs would solve these domains easily.
Spatial relationships are easiest, sequential memory hardest
Skill-level averages separate the models further. Across the four models in the skill table, visual spatial relationship averaged 53.7%, visual figure ground 49.16%, visual discrimination 39.68%, visual form constancy 39.01%, visual memory 38.2%, visual closure 37.51%, and visual sequential memory 32.46%. GPT-5-mini and o4-mini sat near 51–52% on average, while GPT-4o averaged 17.62%.
Domain-level results show where accuracy collapses. "Colours present" averaged 2.89% across the four models, and "layered shapes" 17.95%, while "circle right triangle" reached 88.87% and "inside circles" 83.85%. The spread within a single skill shows that a skill label alone does not predict difficulty; the specific operation the domain requires does.
More objects and ordered answers hurt accuracy
Two failure modes recur across pretrained models. First, accuracy falls steadily as problem size grows, whether size is set by object count, grid size, or required processing steps. Second, in-context examples hurt rather than help, which the authors link to difficulties in handling multi-image context.
Answer format adds a separate constraint. The appendix analysis finds that thinking models such as GPT-5-mini, Gemini, and o4-mini are relatively robust to size variance when the answer is an unordered set, but not when the answer is an ordered list. Producing the right elements in the right order is a distinct challenge from identifying them. Prompting style, by contrast, moved averages only modestly: GPT-5-mini scored 53.63% with concise prompts and 52.93% with verbose prompts, while Qwen scored 13.34% and 14.4% respectively.
Fine-tuning helps, but gains barely transfer
Fine-tuning an open-source MLLM on Percept-V produced considerable gains on the benchmark itself. Those gains only marginally carried over to other related datasets, which the authors read as a limitation in the generalization ability of the learned representations. Ablations further suggest that both the visual representation and the multimodal alignment need fine-tuning for best performance, pointing to limits in the underlying representation and alignment rather than in prompting alone.
A separate memory experiment sharpens the diagnosis. The authors fine-tuned a model to encode images into parametric memory as unique text IDs, then removed the image at test time and queried only the ID. Under that long-term recall constraint, performance collapsed to 1.23%. The authors conclude that MLLMs fail both at maintaining short-term visual representations and at retrieving long-term visual memories.
Perception remains a separate bottleneck
The paper's own interpretation is that these results expose perceptual limitations that must be addressed to build more robust MLLMs. Because Percept-V deliberately minimizes reasoning and knowledge demands, a low score cannot be explained away as a failure of high-level reasoning; it points to how visual information is encoded, held, and used. The 2D-versus-3D comparison in the appendix is consistent with fragility rather than a single missing capability: GPT-5-mini averaged 73.50% on six domains rendered in 2D but 55.33% in 3D, while Qwen moved from 39.8% to 44.5%.
The practical priority follows from the pattern rather than from any single number. The steepest losses appear where many objects must be tracked, where order must be preserved, and where representations must survive without the image. For readers comparing models, full-dataset averages near 41% and skill averages in the 32–54% range are the relevant reference points, not the high scores on a handful of easy domains such as circle right triangle or inside circles.
Sources used in this answer
The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?
On Percept-V, 6,000 program-generated images across 30 domains mapped to TVPS-4 perception skills, state-of-the-art MLLMs averaged 41.28% accuracy, the best model averaged 66.9%, and human performance was very high; accuracy declined as object count grew, and fine-tuning gains barely transferred to related datasets.