How well can models perform reasoning-informed visual editing?

A benchmark that scores reasoning before pixels

Reasoning-informed visual editing (RISE) asks a model to change an image in a way that first has to be inferred: what should the scene look like after a physical reaction, a rule-based transformation, or a viewpoint shift? The paper argues that existing editing models can often render a plausible change when the target is stated explicitly, but struggle when the target state must be derived from visual evidence, outside knowledge, or several reasoning steps. RISEBench++ is built to measure that gap rather than general image quality.

The benchmark organizes RISE into six reasoning dimensions: temporal, causal, spatial, logical, counterfactual, and hybrid reasoning, the last covering multiple reasoning types across multi-turn edits. These split into 12 subcategories and 65 fine-grained task types, so a model's weaknesses can be read at the level of a single ability instead of a coarse category average. The test set contains 1,000 human-annotated cases, released in English and Chinese, and supports single-image, multi-image, and multi-turn inputs. The case counts are uneven by design: 200 each for temporal, causal, spatial, and logical reasoning, and 100 each for counterfactual and hybrid reasoning.

Why one case needs three full scores

Each output is judged on three dimensions. Instruction reasoning asks whether the model understood and executed both the explicit directive and the implicit requirement behind it. Appearance consistency asks whether visual elements unrelated to the instruction survived the edit, which separates models that edit the given image from those that regenerate a scene. Visual plausibility asks whether the modified regions are coherent and free of artifacts, and for temporal, causal, and spatial tasks whether they remain physically coherent.

The scoring rules are not uniform across dimensions. Appearance consistency uses a 1-to-5 scale for temporal, causal, and spatial tasks, whose inputs are visually complex natural images, but a binary 5-or-1 scheme for logical tasks with stylized or synthetic layouts. Counterfactual cases are not penalized for violating physical laws, since implausible outcomes may be the point. Dimension scores are normalized to a 1-to-5 range and rescaled to 100, but accuracy is stricter: a sample counts as solved only when all three metrics reach 5. That definition matters, because a model can look good on averaged dimension scores while failing the all-three-full-score bar that produces the headline accuracy figure.

The strongest model still fails nearly half

The headline result is a ceiling well below saturation. GPT-Image-2.5 Sunburst, the best of the 58 evaluated approaches, reaches 56.6% accuracy. The evaluation pool spans 34 open-source models, 19 closed-source models, and 5 agentic methods, so the shortfall is not an artifact of testing only weak systems. Under the paper's all-three-dimensions-at-5 rule, that means roughly 43% of cases fail at least one of instruction reasoning, appearance consistency, or visual plausibility.

The authors read this pattern as evidence that the bottleneck sits in reasoning before generation rather than in image synthesis. Their reasoning is that models often produce visually plausible, appearance-consistent results when the desired change is spelled out, and fail when the target state must first be inferred. That is the authors' interpretation of the aggregate pattern, not a separately isolated causal test; the benchmark's dimension scores are what make the interpretation plausible.

Explicit rules are easier than hidden patterns

The subcategory results show a clear imbalance. Leading models exceed 60% accuracy on both spatial relation and spatial structure, indicating relatively strong handling of object relationships and scene organization. Logical reasoning splits sharply: explicit rule deduction is difficult but achievable, while implicit pattern induction stays low, with even the strongest closed-source result reaching only 31.2%. Applying a stated rule appears easier than discovering a latent regularity from the image and turning it into a precise edit.

Counterfactual reasoning follows a different pattern. GPT-Image-2.5 Sunburst reaches 71.4% on counterfactual imagination and 73.3% on anomaly correction and generation, suggesting that strong models can both detect violations of familiar rules and deliberately construct outcomes that contradict them. At the task level, pattern prediction stays at or below 25.6% across the seven models analyzed in detail, and errors show up in symbol positions and orientations. The paper notes that appearance consistency and visual plausibility alone therefore do not guarantee correct instruction reasoning.

An agent that plans, executes, and checks

RISE-Agent is the paper's training-free attempt at a solution, built from three roles. A planner infers the intended target state, calling on web search when domain knowledge is needed and on seven code-based tools covering arithmetic, date calculation, grid and graph shortest-path search, Sudoku solving, coordinate geometry, and matrix transformation. An executor then realizes the plan either with a generative editor or with a toolkit of 12 deterministic operations such as line drawing, region recoloring, and content relocation, which leave untouched pixels unchanged. A verifier scores the candidate and chooses to pass, refine, or replan, within fixed refinement and replanning budgets.

The component comparison shows the loop matters. Adding verifier-guided refinement raises the overall score to 48.0% and gives the best result in five of the six reasoning categories. The gain is smallest on spatial tasks, which the authors attribute to the verifier's limited spatial understanding. RISE-Agent also reaches 47.2% on explicit rule deduction, above every evaluated closed-source model on that subcategory, and its overall accuracy of 65.9% in the reported comparison table exceeds the other agentic methods listed there. These are results for one agentic configuration, not evidence that agents in general close the gap.

Multi-turn edits accumulate visual decay

Hybrid cases expose a failure mode that single-turn scores hide. In a sequential example, a clock is advanced by one hour, rotated 180 degrees, and then its minute hand is rotated clockwise by 3,600 degrees; each intermediate result must stay both semantically correct and visually stable for the next edit. Some models preserve the scene reasonably well across turns, but Qwen-Image-Edit-2509 and HunyuanImage-3.0 show cumulative visual degradation, with later outputs progressively losing structural integrity and fidelity.

The paper's reading is that errors in multi-round reasoning-informed editing accumulate at two levels: the reasoning itself and the repeated visual regeneration. That makes long-horizon consistency a separate challenge from solving any single edit correctly. Because this observation comes from qualitative examples rather than a dedicated multi-turn accuracy breakdown, it should be treated as a documented failure pattern, not a quantified ranking of models on multi-turn robustness.

Where the 56.6% ceiling applies

The practical priority follows from the subcategory split rather than from the aggregate score. If a task requires discovering an implicit visual rule, the relevant reference point is the 31.2% best closed-source result on implicit pattern induction and the 25.6% task-level ceiling on pattern prediction, not the 56.6% overall figure. If a task requires applying an explicit rule to a structured layout, RISE-Agent's 47.2% on explicit rule deduction is the more informative comparison, and it beats the closed-source models there. For tasks that lean on familiar world knowledge, such as counterfactual imagination and anomaly correction, leading models already exceed 70%.

Two boundaries qualify these numbers. Accuracy is an all-three-dimensions-at-5 measure, so it is not interchangeable with the rescaled 100-point dimension scores reported elsewhere in the paper. And the benchmark's own judging relies on an LMM-as-a-judge pipeline validated against human assessment, with judge-model agreement varying by dimension; the reported judge comparison shows, for example, a Spearman correlation of 0.81 for GPT-5 on the overall score. The most defensible takeaway is that progress in RISE should be tracked per reasoning subcategory, because the gap between explicit rule application and implicit pattern discovery is larger than the gap between the best model and the rest.

Sources used in this answer