Do cognitive expert language models align better with matching brain systems?

One model for all regions hides specialization

Typical model–brain alignment studies fit a single language model to many brain regions and then average its predictive performance across them. That design treats the brain as one target, even though the paper notes that it is organized into partially specialized systems supporting sensory processing, abstract meaning, spatial navigation, calculation, reasoning, and thinking about other people. The study asks a narrower question: if a model is pushed toward one cognitive domain, does it align better with the brain system dedicated to that domain?

The authors answer this with an encoding model. Participants listen to or read language while fMRI records their brain activity, and the same language is passed through the model to obtain internal representations. A linear mapping, fitted by ridge regression on training stimuli, predicts held-out brain responses, and alignment is the Pearson correlation between predicted and actual responses. A model that predicts a region well is taken to capture features of the stimulus that the region represents.

Six experts per base model, two ways

An expert here is a variant of a base model specialized for one cognitive domain, and a family is the set of six experts derived from one base model under one intervention. The authors built experts in two ways. Prompting places a fixed system prompt naming the domain in the input while leaving the weights untouched. Fine-tuning trains LoRA adapters on instruction–response pairs from the domain, with checkpoints selected under a guard that keeps general-text perplexity close to the base model's, so specialization does not cost general language ability.

Both interventions were applied to three base models—LLaMA-3-8B, Qwen3-8B, and Phi-4-14B—producing one prompted and one fine-tuned family per base model, each with an expert for sensory, spatial, numerical, reasoning, social, and abstract-concept domains. Before any brain data were used, each expert had to show behavioral specialization. Fine-tuned experts were checked by perplexity on held-out domain text, where each expert improved most on its own domain among the six, with mean own-domain improvements of 22% to 40% over the base model. Prompted experts were checked on what they generated: continuations of neutral story openings, scored against domain lexicons with instruction words removed and rated by a separate LLM judge blind to the prompt. On the lexicon measure, the matched expert produced 3.7, 1.9, and 3.1 more own-domain words per 100 than the base model for LLaMA, Qwen, and Phi-4 respectively.

Regions come from meta-analysis, not hand-picked spots

Brain regions for the six domains were derived from Neurosynth, an automated meta-analysis of the fMRI literature. For each domain the authors combined maps for a small set of related terms and thresholded the result into a binary mask on a common template, producing a network of several areas that need not be contiguous. These masks were projected into each subject's functional space. The Pereira dataset instead supplies a parcellation into five functional networks—language, multiple-demand, default-mode, visual, and auditory—and each network was assigned to one domain in place of the Neurosynth regions.

Alignment was evaluated on three fMRI datasets: three densely sampled subjects listening to spoken narratives (LeBel), the 49-subject English cohort of Le Petit Prince (LPP), and ten subjects reading passages one sentence at a time (Pereira). The first two are naturalistic narratives that engage varied cognitive processing across many regions within the same comprehension task; the third tests whether the effect survives a more controlled sentence-by-sentence format.

Displacement measures rank, not raw accuracy

The primary statistic is displacement. For each brain system, the six experts of a family are ranked by how well each predicts activity inside that region relative to outside it: each expert's encoding accuracy, with the family mean across experts subtracted at every voxel, is averaged inside and outside the region, and the six experts are ranked on the difference. Displacement takes the rank of each region's own expert (1 = best), averages it over regions, and subtracts it from the chance rank of 3.5, the midpoint of 1 to 6. Zero is chance, and positive values mean matched experts rank nearer the top.

Because six experts can be assigned to six regions in exactly 720 ways, the null distribution is exact, and the p-value is the fraction of relabelings scoring at least as high; on Pereira's five networks there are 120 relabelings and a chance rank of 3.0. The authors also report Δr, the same comparison in correlation units: the expert's mean encoding accuracy inside the matching region minus that of the other five experts, averaged over regions and tested across subjects. Displacement keeps only the rank of an expert's advantage inside the region over outside it, whereas Δr keeps the size of that advantage inside the region. These are related but not interchangeable measures.

Matched experts rank nearer the top in every family

Consistent with the hypotheses, each expert's representations aligned more closely with the brain system most associated with its matching domain than did the other experts. The effect held under both prompting and fine-tuning, across all three base models and all three fMRI datasets. The authors report that the effect also survives a hyperparameter sweep: put through the same sweep, relabeled experts reached a mean displacement of +0.31, against +0.55 for the actual experts (p = .0002).

The result matters for domains whose brain systems lie largely outside the language network, according to the discussion. In other words, the alignment shift is not confined to regions that a language model would be expected to capture anyway. The paper frames this as evidence that specializing a model moves its representations toward the corresponding brain system rather than simply improving language-region prediction.

Content-free interventions do not reproduce the shift

To test whether alignment is specific to cognitive content, the authors built five control families with the same construction procedures but no cognitive content: random-LoRA adapters matched in norm to the cognitive ones; seed-only adapters trained on one pooled dataset mixing all six domains, differing only in sampling and initialization; random prompts making the model an expert in fields such as zebra striping anatomy or vintage typewriter restoration; surface-form adapters trained on style-transformed text such as leetspeak and pirate English; and keyword conditions using each domain's bare vocabulary with no instruction.

Each control family was passed through the same encoding pipeline. Because a control family has no true expert-to-region mapping, scoring it under every possible assignment of its members to the regions yields a null distribution of displacement centered on zero. The authors then asked whether the expert families exceed that distribution, comparing each cognitive result with a content-free control and with the control sharing its intervention. The reported outcome is that non-cognitive and surface-level interventions do not result in comparable alignment, which supports the interpretation that the shift is tied to cognitive domain content rather than to the act of intervening on the model.

Specialization shifts regions while aggregate accuracy holds

Specializing models shifted regional alignment while leaving aggregate prediction accuracy largely unchanged. This is the paper's most consequential comparison for readers who evaluate models by a single summary number: a model can look equivalent to another on overall brain prediction while distributing its accuracy differently across regions. The authors conclude that summarizing alignment across regions may obscure regional differences in performance for specific models.

The discussion adds an interpretation the authors draw from the prompting result: because a prompt changes no weights yet works about as well as fine-tuning, specialization may amplify a representational structure already present in a model trained only on text. That is the authors' explanation, not a separately measured mechanism. The practical implication follows from the paper's own variables: when comparing models on brain alignment, the choice of region and the matching between model specialization and brain system matter, and an aggregate score alone can hide the difference.

What the six-domain design can and cannot settle

The claim is bounded by the domains tested and the regions used. The six domains are sensory, spatial, numerical, reasoning, social, and abstract processing, and the regions come from Neurosynth term maps or, for Pereira, a five-network parcellation with one network assigned per domain. The displacement statistic compares each expert only with the other five in its family, so it measures relative ranking within that set rather than absolute alignment with a brain system. The exact permutation tests over 720 relabelings (120 on Pereira) mean no displacement p-value depends on a sampling seed, but they do not extend the claim beyond the tested models, datasets, and domain definitions.

The most useful takeaway is a comparison rule rather than a general verdict: within this design, a model specialized toward a cognitive domain is the better predictor of the matching brain system, and that advantage appears in regional rank even when aggregate accuracy does not move. Readers comparing models on brain alignment should therefore ask which region a model is being scored against and whether the model was specialized toward that domain, because the paper's evidence shows those choices change the ranking while a single averaged score can conceal it.

Sources used in this answer