Do multimodal LLMs align with human and brain responses to cartoon movies?

Three open models against human raters

The paper asks whether multimodal large language models (MLLMs) can stand in for human annotators of social and emotional content in naturalistic movies. The comparison is concrete: continuous human ratings of Theory of Mind (ToM) and affective valence, collected at 4 Hz on a −10 to +10 joystick scale, versus frame-by-frame ratings generated by three open-weight models. ToM here means reasoning about characters' beliefs, intentions, and desires; valence means the positive-to-negative dimension of emotional experience.

The models span two generation paradigms. LLaDA-V (8B) is diffusion-based, producing text by iteratively denoising masked tokens; Qwen3-VL-8B-Instruct and Gemma 3 12B-IT are autoregressive, instruction-tuned models that generate tokens sequentially. All received the same prompt template and rating instructions and saw sampled frames without preceding context in the baseline condition.

The stimuli were three animated shorts: The Present (3.3 min), Despicable Me (10 min), and Partly Cloudy (5.6 min, silent). Human ratings came from separate cohorts of healthy young adults (34 ToM raters, 42 valence raters), while the fMRI data came from different participants: 87 for The Present and 83 for Despicable Me from the Healthy Brain Network, and 82 for Partly Cloudy from OpenNeuro. That split matters for reading every neural result below.

Valence aligns better than mentalizing

Averaged across all three models and movies, human–model agreement was r = 0.434 for valence versus r = 0.265 for ToM. Seven of the nine model–movie combinations fell above the identity line, meaning stronger alignment with human valence than with human ToM judgments.

The asymmetry was not uniform. Gemma 3 showed the widest gap, with the highest mean human agreement for valence (r = 0.485) but the lowest for ToM (r = 0.186), including a near-zero, slightly negative correlation on Despicable Me (r = −0.05). LLaDA-V and Qwen3-VL showed a milder version of the same pattern.

The authors attribute the gap to what is visible in a frame: facial expressions and scene composition offer accessible emotional cues, whereas mental-state judgments often need information beyond the current frame. They also note that models saw only visual frames, so missing dialogue may matter; on dialogue-rich Despicable Me, ToM alignment was weakest while valence alignment stayed high. That explanation is the authors' interpretation, not a separately tested mechanism.

Models agree with each other more than with people

Inter-model agreement exceeded human–model agreement overall. The models also used the rating scale differently: human ratings clustered in a narrow central band of the −10 to +10 range, while LLaDA-V and Qwen3-VL tended to assign higher scores than humans, especially for ToM. Gemma 3's distributions came closest to human ones, though differences remained.

So a high correlation between a model and humans does not mean the model reproduces human rating levels. The reported correlations measure temporal alignment — whether the two time series rise and fall together — not whether they occupy the same part of the scale. A model can track the shape of human ratings while systematically overstating intensity.

The paper also reports a least-squares fit suggesting a negative trend between ToM and valence alignment (y = −0.49x + 0.56, R² = 0.38), but flags it as cautious reading: it rests on only nine non-independent observations.

Preceding scene descriptions help unevenly

In a second condition, the models first generated descriptions of earlier frames, and those descriptions were added to the rating prompt alongside the current frame. The number of descriptions was held fixed (five for The Present and Despicable Me, four for Partly Cloudy) and spread across lookback windows of 4, 8, 12, and 16 seconds.

Qwen3-VL and Gemma 3 generally improved or held steady, particularly at intermediate lookback windows of roughly 8–12 seconds. Gains were more consistent for valence, and both measures benefited in Partly Cloudy. LLaDA-V showed little improvement across context conditions.

One exception is instructive: Gemma 3's ToM alignment dropped markedly under the 4-second context condition on The Present. Inspecting early frames (about 8–11 seconds) showed stronger mental-state attributions despite low human ToM ratings. With context, the model referred to earlier frames to interpret the current scene; without it, the model expressed uncertainty about lacking context. The authors suggest preceding descriptions may sometimes encourage over-interpretation of ambiguous scenes. Because description counts were fixed, longer windows tested how far back sampled context reached, not how much context was supplied.

Model ratings predict held-out brain responses

Each rating source — human, LLaDA-V, Qwen3-VL, Gemma 3 — was used separately in subject-specific ridge-regression encoding models to predict BOLD responses across 114 regions of interest (100 cortical regions from the Schaefer atlas, 14 subcortical from the AAL atlas). Ratings were shifted by a fixed 4-second hemodynamic delay, and performance was measured as the Pearson correlation between predicted and held-out responses under nested temporal cross-validation with 16-second segments.

Encoding performance varied substantially by movie, measure, and source. The Present showed stronger positive correlations and more widespread significant encoding, especially for MLLM ratings, concentrated in visual, dorsal-attention, and default-mode networks, with additional salience/ventral-attention involvement. Default-mode involvement was especially extensive for valence.

The Present also produced the headline count: more regions reached significance for MLLM ratings than for human ratings. That did not generalize. For Partly Cloudy valence, more regions reached significance for human ratings than for any individual MLLM. Because each source was tested against zero separately, these counts describe where encoding was reliably above chance for that source; they do not establish that one source predicted brain responses better than another.

Shared brain regions are mostly model-only

Spatial overlap maps show where multiple rating sources each had significant encoding. The most extensive convergence appeared in The Present, and most of it was among MLLMs rather than between models and humans.

For ToM in The Present, two-source overlap involved visual and dorsal-attention regions plus parts of the default-mode and salience/ventral-attention networks, and was almost entirely between Qwen3-VL and Gemma 3. Three-source overlap concentrated in visual regions, where all three MLLMs converged, and in bilateral posterior cingulate/precuneus, where human ratings overlapped with Qwen3-VL and Gemma 3. For valence, three-source overlap spread broadly across the default-mode network plus visual, dorsal-attention, and salience/ventral-attention regions, but stayed confined to the three MLLMs except in the right posterior cingulate/precuneus, where human ratings were one of the three significant sources. No four-source overlap occurred in The Present.

Overlap was more spatially restricted in the other two movies, but human ratings made up a larger share of it. In Despicable Me, ToM showed a focal four-source overlap in the right temporo-occipito-parietal salience/ventral-attention network — the only region significant for human ratings and all three MLLMs. Human valence ratings in Despicable Me yielded no significant regions at all.

What these annotations can and cannot claim

The paper's own framing is that MLLMs offer scalable, brain-predictive annotations of social and emotional content, while alignment with humans varies across models, measures, movies, and temporal context. The practical reading is narrower than "models understand social and emotional content": the evidence supports temporal tracking and above-chance neural prediction, not human-like social understanding.

Several limits sit close to those results. Predictive associations establish neither human-like social understanding nor specificity to ToM or valence, since both measures may co-vary with broader factors such as character presence, visual motion, or arousal. Individual frames and scene descriptions may miss contextual and auditory cues. Human ratings came from young adults while imaging participants were largely children and adolescents, so the human reference is an adult one. And the lag-optimized correlations that headline the behavioral comparison were selected and evaluated on the same data, making them upwardly biased and explicitly exploratory; the paper reports that lag adjustment changed correlations by at most 0.09 (median 0.025) relative to unshifted values.

Where alignment is most trustworthy

The most defensible priority from this study is measure- and movie-specific rather than model-general. Valence is the stronger target: mean human–model agreement of r = 0.434 across models and movies, with seven of nine model–movie combinations above the identity line. ToM is the weaker target, and the weakest single case is Gemma 3 on dialogue-rich Despicable Me (r = −0.05).

The Present is the movie where MLLM ratings carried the most widespread significant encoding and the most cross-source overlap — but that overlap was largely model-to-model, so it is evidence of convergence among models, not of shared human–model structure. Where human and model encoding did coincide, it centered on visual, dorsal-attention, and posterior default-mode regions, with the right posterior cingulate/precuneus recurring as a human-inclusive site.

For anyone weighing MLLM annotations as a substitute for human rating, the paper's own split is the useful one: use them where the target is affective valence and the stimulus supplies visible emotional cues, treat ToM annotations as the weaker and more model-dependent case, and do not read agreement among models as validation against people.

Sources used in this answer