Testing prediction, not just observation
Most brain–vision-model alignment work asks how well a model's internal representation of a seen image or video matches cortical responses to that same stimulus. This paper starts from a different premise: the visual cortex does not only respond to what arrives, it also predicts what comes next. If that is true, a model built to generate the next frames should match cortical activity better than a model built to represent the frames already shown.
The authors test this with human fMRI recorded while ten subjects watched 1,102 naturalistic videos of 3 seconds each, 1,000 used for training and 102 for testing. Responses are analyzed over 7,894 vertices of the visual system, grouped into five divisions: Primary Visual, Early Visual, Dorsal Stream, Ventral Stream, and MT+ Complex. Only vertices with a noise ceiling above 10% enter the analysis, so the comparison is restricted to cortical locations where responses are reliable enough to model.
Two models that differ in temporal order
The comparison uses two video diffusion models. A conventional non-autoregressive model denoises all frames of a video simultaneously, which the paper describes as disregarding the causal order of time. The autoregressive (AR) model instead generates future frames conditioned on the observed video, so its internal state must represent how the stimulus unfolds over time.
This design matters because the two models share a base architecture, letting the authors compare representations for future video generation against representations for observed video reconstruction without confounding them with an entirely different model family. The AR model is first analyzed on its own, then compared with its non-AR base.
Future generation aligns better than the observed video
In the within-model analysis of the AR model, the representations used for future video generation align better with the visual cortex than the representations of the observed video — even though both the model and the brain receive that same observed video. The paper further reports that the observed tokens add no unique contribution beyond future video generation in any of the five divisions, which the authors treat as confirmation that future generation is the key to the alignment.
The cross-model comparison points the same way: future video generation in the AR model aligns better with the visual cortex than observed video reconstruction in the non-AR base model. The reported effect sizes are modest — the figure shows unique contributions on the order of 0.07 for observed video and 0.09 for future generation in R²/NC units — so the claim is a relative ordering between representations, not a claim that either model explains cortical responses well in absolute terms.
A split across the visual hierarchy
The more specific result is where each representation aligns. The unique contribution of observed video reconstruction is concentrated in lower-order visual cortex, the divisions that process simple features such as edges, contours, and textures. The unique contribution of future video generation is concentrated in higher-order visual cortex.
The authors connect this split to the denoising schedule. Future generation aligns best at the noisiest step and at an early-to-middle layer, a state the paper calls uncommitted: as denoising proceeds, the future video settles into one of many possible futures. Observed video reconstruction instead aligns best at a less noisy step, where fine details are refined. The authors read this as indirect evidence that the visual cortex, especially higher-order cortex, carries future-oriented computation, and they note consistency with Bayesian accounts in which neural populations encode a range of possible stimuli rather than a single prediction.
Human preferences follow the neural layer profile
The paper adds a behavioral test rather than stopping at neural alignment. Using a method that amplifies the contribution of an individual layer during future video generation, the authors asked humans to compare generated videos. Participants preferred videos produced by amplifying the layers that aligned better with the visual cortex.
This links the layer-wise alignment profile to a human judgment about the generated video, but the authors are explicit that it is not a causal account of how neural alignment produces preference. The behavioral result shows that the alignment ranking has a perceptual correlate; it does not show that alignment causes the preference.
What the 3-second design can and cannot show
The analysis framework feeds a 3-second observed video and generates the 1-second future that follows. That window bounds the claim: the paper does not establish how the brain represents past and future over longer contexts or longer horizons. The mapping also uses one response per video at each cortical vertex, so the time course of the cortical response is not compared with the AR model's internal representation over time.
A further comparison in the appendix places future video generation against a wide range of vision models. Self-supervised video models aligned best with the visual cortex overall, but the gap between future video generation and those models narrowed toward higher-order visual cortex — a pattern the authors offer as further indirect evidence for a predictive mechanism in higher-order cortex, not as a demonstration that the AR model outperforms self-supervised models across the cortex.
The hierarchy, not the overall score, is the finding
The paper's most useful contribution is not a single alignment number but a division of labor: observed-video reconstruction tracks lower-order visual cortex, while future-frame generation tracks higher-order visual cortex, and the future-generation representation beats the observed-video representation in both the within-model and cross-model comparisons. That ordering is what makes the predictive framing testable rather than merely plausible.
For readers weighing this evidence, the practical priority is to look at which cortical division a representation aligns with, not only at aggregate alignment. The paper's own comparison with self-supervised video models shows that aggregate rankings can reverse depending on the cortical level examined, and its 3-second input with a 1-second future horizon means the higher-order advantage is established for short naturalistic clips, not for longer predictive contexts.
Sources used in this answer
Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video
Representations for future video generation in an autoregressive video diffusion model align better with human visual cortex fMRI responses than representations of the observed video, with future generation concentrated in higher-order visual cortex and observed reconstruction in lower-order visual cortex.