Beyond Object-Level Alignment: Do Brains and DNNs Move the Same Way?

Executive Summary

TL;DR: Most AI-neuroscience research asks if the brain and a model "see" the same thing. This paper asks a deeper question: do they "move" the same way? By introducing the Naturality Violation Score (NVS)—a metric based on category theory—the authors demonstrate that alignment is selective. While an AI might predict brain activity well, it may fail to preserve the same transformations (like changing color vs. changing category). The study reveals a "hierarchy crossover": early brain regions and shallow AI layers agree on visual changes (luminance), while high-level regions and deep AI layers agree on semantic structural changes (animacy).

Background Positioning: This work moves beyond "SOTA-chasing" on Brain-Score. It provides a diagnostic tool to pinpoint why a model is or isn't brain-like by shifting focus from static objects to dynamic transformations.

The Motivation: When Geometry Isn't Enough

Current metrics like Representational Similarity Analysis (RSA) or Canonical Correlation Analysis (CCA) are "static." They tell us if Stimulus A and Stimulus B are positioned similarly in two vector spaces. However, they ignore the edges—the morphisms.

Imagine two systems that both categorize "Dogs" and "Cats" correctly. In System A, turning a cat into a dog is a simple shift in texture. In System B, it’s a fundamental structural reorganization. Standard metrics might call these systems "aligned" because the end-state coordinates are similar, but they operate on entirely different logic. The authors argue that true alignment requires Naturality: the two systems should commute across a transformation square.

Methodology: The Naturality Square

The core innovation is the Naturality Violation Score (NVS). The authors use a "cospan" framework involving three spaces:

1. B (Brain): fMRI voxel patterns.
2. M (Model): DNN layer activations.
3. W (World Model Proxy): An external space (like CLIP or DreamSim) that defines what a "change" (e.g., an increase in animacy) actually looks like.

The Commutativity Test

If you take a stimulus, change it according to a "World" rule (like making it more animate), and then translate that brain activity into model space, does it match what happens if you translate first and then apply the change in the model?

This square is evaluated via linear translators ($\eta ,{\eta }^{\prime }$). Any gap in the square is recorded as a violation.

Key Insights from Experiments

1. The Synthetic Proof

In a controlled 5-factor world (x, y, scale, theta, color), the authors proved that standard metrics like CCA could not distinguish between a model that only understood position and a model that only understood color. NVS, however, cleanly separated them by testing specific "axis-resolved" transformations.

2. The Hierarchy Crossover in Human fMRI

Using the GOD (Generic Object Decoding) fMRI dataset, the authors found a striking pattern across 5 subjects and 3 vision DNNs (AlexNet, ResNet-50, ViT):

• Low-level axes (Luminance, Spatial Frequency): Alignment peaked in V1 and shallow layers.
• High-level axes (Animacy, Real Size): Alignment peaked in the Higher Visual Cortex (HVC) and deep layers.
• The Animacy Factor: Animacy was the most preserved transformation across all tested systems ($NV{S}_{animacy}=0.39$), significantly lower (better) than any other axis.

Strategic Takeaways & Limitations

Critical Analysis: The most profound finding is that alignment is selective. A model can be "brain-like" for textures but "non-brain-like" for semantics. This suggests that the quest for a single "Brain-Score" may be misguided. We should instead be looking for a "spectrum of alignment."

Limitations:

• Proxy Dependency: NVS relies on the "World Model" (W) being accurate. If CLIP doesn't understand a specific transformation, NVS can't test it.
• Linearity Assumption: The study relies on the "Linear Representation Hypothesis." While often true for modern LLMs/VLMs, non-linear transformations might exist that this framework misses.

Conclusion

This paper reframes the Brain-DNN relationship from a comparison of states to a comparison of processes. By proving that the brain and DNNs share a specific hierarchy of transformations—shifting from pixels to semantics—it settles the debate on whether deep networks capture biological structural logic, while providing the tools to measure exactly where they diverge.