From Plato to Aristotle: Calibrating the True Alignment of Neural Representations

TL;DR

Is AI converging toward a single "Platonic" model of reality? This paper argues that much of the evidence for this convergence is actually a statistical illusion caused by model size. By introducing a rigorous null-calibration framework, the authors debunk global spectral convergence and propose the Aristotelian Representation Hypothesis: AI models are not converging in global geometry, but in their local neighborhood relationships.

The Scaling Illusion: Why Width and Depth Lie

In the quest to understand if different AI models (like Vision Encoders and LLMs) "see" the world similarly, researchers have relied on representational similarity metrics. However, this paper identifies two massive "confounders" that make bigger models look more similar than they actually are:

1. The Width Confounder: In high dimensions ($d$), even random, independent noise can appear correlated. As models get wider, their raw similarity scores (like CKA) naturally drift upward, even if there is zero actual signal.
2. The Depth Confounder: When we compare two deep models, we usually look for the most similar pair of layers. The deeper the model, the more pairs we check—increasing the chance of finding a "spuriously similar" pair by pure luck.

The Solution: Null-Calibration

To fix this, the authors introduce a metric-agnostic calibration framework. The logic is elegantly simple:

• Permute: Shuffle the data labels to break any real correspondence between two models.
• Measure: Calculate the similarity on this "nonsense" data to see what "chance" looks like for that specific model pair.
• Calibrate: Score the real similarity only by how much it exceeds this "chance" threshold.

They also introduce aggregation-aware calibration, which ensures that the process of picking the "best" layer pair doesn't inflate the final result.

Methodology: Redefining Convergence

The authors put the "Platonic Representation Hypothesis" (which suggests universal convergence across modalities) to the test. They analyzed 204 vision-language model pairs (including LLaMA, DINOv2, and CLIP) using:

• Spectral Metrics: (CKA, CCA) - Measuring global subspace alignment.
• Neighborhood Metrics: (mKNN) - Measuring local "who is near whom" relationships.

Note: The authors provide Algorithm 1 and 2 in the paper to implement scalar and aggregation-aware calibration, ensuring results are comparable regardless of embedding dimension or layer count.

Real Results: The Aristotelian Shift

When the metrics were calibrated, the "Platonic" trend for global metrics (CKA) essentially vanished. Models aren't becoming more globally similar as they scale; they just have more dimensions to hide spurious correlations in.

However, local neighborhood metrics (mKNN) survived the calibration.

Key Discovery:

• Topological Agreement: Models agree on which data points belong together in a cluster.
• Metric Disagreement: Models do not agree on the exact distances between those points. Small-bandwidth RBF kernels (which are hyper-sensitive to distance) showed zero alignment after calibration.

This led to the Aristotelian Representation Hypothesis: convergence happens in relatives (neighborhoods), not in absolute Platonic forms.

Critical Insight & Conclusion

This work serves as a necessary "statistical health check" for the AI community. It demonstrates that as we scale, our evaluative tools must also scale in sophistication.

Takeaway: If you are measuring model similarity, you cannot trust raw scores. You must calibrate against a null distribution to ensure your "SOTA alignment" isn't just a byproduct of having a high-dimensional Latent Space. The future of representational research lies in understanding the local topology—the shared "Aristotelian" relationships—rather than hunting for a global "Platonic" isomorphism that might not exist.

Limitations: While permutation calibration is robust, it assumes data samples are exchangeable. In time-series or highly clustered data, more complex "restricted permutations" would be required to maintain validity.