Do generative model priors align with human judgments of image naturalness?

Why raw losses fail and paired differences work

The paper asks whether the internal probability landscape of a generative model — its prior over natural images — resembles the regularities that shape human judgments of naturalness. The obvious probe is the model's own prediction error: if an image looks unnatural to a person, does the model also assign it a higher loss? Tested directly, this fails. Correlating single-image losses with single-image unnaturalness ratings gave 25-model median Pearson correlations of .236 and −.090 on Thatcher faces (original and modified images) and −.145 and −.203 on illumination scenes. Raw loss is dominated by scene content and visual complexity, so it says little about whether a specific relational rule was broken.

The study's central move is to subtract. For each stimulus pair, the authors compute the loss change between an original image and a modified version of the same scene, u = L(modified) − L(original), and compare it with the human paired unnaturalness effect h, the drop in ratings caused by the same modification. Because both images share content, this paired difference isolates sensitivity to the targeted violation. Under this design, all 25 models correlated positively with human paired judgments in both tasks, with median correlations of .747 on Thatcher faces and .334 on illumination scenes.

Content-preserving edits isolate relational violations

The manipulations are designed to change relations while leaving low-level statistics largely intact. In the Thatcher task, 140 public-domain face scenes were edited by vertically reflecting the eye and mouth regions in place, using facial landmarks, and then shown upright and inverted, producing 280 original–modified pairs. In the illumination task, 96 rendered scenes each received three interventions — cast-shadow swap, reflection swap, and light-direction swap — for 288 pairs. Shadows and reflections were exchanged by mirroring the floor region while objects stayed fixed; in the light-direction condition the central object was mirrored while flanking objects and floor shadows were unchanged.

This matters because a simpler control behaves differently. When a cast shadow is deleted entirely, the scene becomes physically impossible but also less complex, and loss changes were uniformly negative across all models: the corrupted scene looked more probable. Preserving low-level statistics under the swap design removes that artifact, allowing more capable architectures to show positive loss penalties for the physical inconsistency. The paired, content-preserving design is therefore not a cosmetic choice; it is what makes the comparison with human judgments interpretable.

Human ratings set the comparison target

Human data came from online psychophysical experiments using a 5-point naturalness scale from 1 (very unnatural) to 5 (natural), with the slider initialized to a random position to avoid anchoring. Participants were instructed on the specific manipulation types — Thatcher, shadow swap, reflection swap, light direction — so that ratings reflected structural plausibility rather than photographic aesthetics. After response-consistency screening, 143 observers were retained for the Thatcher task and 105 for illumination, with each item receiving an average of 35.8 and 26.2 ratings respectively.

Two human measures follow from item-mean ratings R(x): single-image unnaturalness a(x) = 5 − R(x), and the paired effect h = R(original) − R(modified). Only item-level aggregates enter the model analyses. The distinction between these two measures is the same distinction that separates the failed raw-loss analysis from the successful paired analysis, so the model and human sides are compared on matching quantities rather than on loosely related scores.

Models reproduce selective human sensitivities

The models do not merely flag any change. They reproduce the Thatcher effect: sensitivity to the eye-and-mouth inversion is pronounced for upright faces and deteriorates when the whole face is inverted, matching the classic orientation dependence of human face perception. On physical scenes, model responses follow human tolerance patterns tied to object geometry — observers readily detect shadow and reflection inconsistencies for objects with distinctive directional landmarks, such as a teapot, but tolerate errors for shapes lacking prominent spatial asymmetries. Generative losses tracked this geometric selectivity across shadow and reflection rotations and lighting directions.

The authors read this as evidence that models rely on statistical correspondence cues resembling human perceptual heuristics rather than acting as objective detectors of physical violations. That interpretation is theirs, but it is anchored in a concrete divergence: models responded much more weakly than humans to reflection hue shifts. Humans treat an impossible reflection color as a decisive violation, whereas generative losses penalize it only as a modest color difference relative to geometric misalignment.

Alignment survives encoder and quality controls

A natural objection is that any representation sensitive to image change would correlate with human ratings. The paper tests this with two baseline families: feature distances from frozen vision encoders spanning 24M to 8.1B parameters, including ResNet-50, CLIP, DINOv2/v3, SigLIP 2, PE-Core, Qwen3-VL-Embedding, and Jina CLIP v2, with the best-performing layer selected post hoc in the encoder's favor; and two five-metric image-quality batteries, full-reference (PSNR, SSIM, VIF, LPIPS, DISTS) and no-reference (NIQE, BRISQUE, HyperIQA, MUSIQ, LIQE).

Bidirectional partial correlations show that generative scores retain human-aligned signal after controlling for these baselines, and that the baselines retain less after controlling for generative scores. The advantage is largest for upright faces, cast shadows, and light direction. Under face inversion, residual correlations weaken for both generative models and baselines, consistent with the overall drop in perceptual sensitivity. For surface reflections, both sides keep positive residual associations, indicating that generative and encoder representations capture partly distinct aspects of reflection naturalness. The authors note that fixed feature distances measure the magnitude of change rather than the direction of degradation, so these baselines are a diagnostic reference rather than an upper bound.

Sensitivity and alignment peak at different times

Across models, overall sensitivity to violations broadly covaries with human alignment, but the relationship is condition-dependent. Positive associations were robust for inverted faces (r = .723, 95% CI [.118, .923]) and cast shadows (r = .810, [.424, .960]); the upright-face estimate was positive but uncertain (r = .667, [−.215, .904]), and associations were absent or negative for light direction (r = .051) and surface reflections (r = −.371).

More tellingly, the two properties separate within a single model's denoising schedule. As the schedule position moves from 0 to 1, human alignment and violation sensitivity peak at different points, with alignment peaking earlier than sensitivity across the 25 models. The paper treats this as evidence that human-like naturalness judgments dissociate from generic violation detection: a model can become more sensitive to perturbations without becoming more human-aligned. Exploratory scaling comparisons point the same way — whole-task alignment rose with model size, but for surface reflections larger models grew steadily more sensitive with no improvement in human alignment.

Where the claim holds and where it stops

The evidence supports a specific claim: native loss landscapes of current generative models capture distinct aspects of human naturalness perception, not a general equivalence between model probability and human judgment. The strongest alignment appears on faces, where top generative models markedly outperform even layer-optimized encoders, and the weakest on reflection color, where human and model weightings diverge. The authors also report that the orientation penalty appears in ImageNet-trained JiT models without dedicated face training and scales with capacity, and that generative losses track naturalness variation within upright faces alone (median r = .493).

The scope is bounded by design. The relational intervention framework minimizes low-level confounds at the cost of scene diversity, so extending it to unconstrained natural imagery without introducing artifacts remains open. Because off-the-shelf models are compared, the analysis cannot fully separate core generative objectives from differences in scale, architecture, or post-training. The practical priority that follows is not to treat lower generative loss as a general naturalness score, but to use paired, content-preserving loss differences as a diagnostic — and to read alignment and sensitivity as separate axes, since the schedule analysis shows they can move apart even when overall sensitivity looks like a good proxy.

Sources used in this answer