[ICLR 2025] The Geometry of Systematicity: Why Compositional Generalization Demands Linearity and Orthogonality

Executive Summary

TL;DR: Why can a model recognize a "cat on a mat" but fail to understand a "mat on a cat"? This paper proves that for a vision model to truly master such recombinations, its internal "mental map" must be a linear dictionary. It demonstrates that linear factorization (adding concept vectors) and cross-concept orthogonality are not just elegant features—they are mathematical necessities for any model trained via gradient descent to achieve stable compositional generalization.

Background Positioning: This work moves beyond empirical observation (the Linear Representation Hypothesis) to provide a theoretical foundation. It situates itself as a bridge between high-dimensional geometry and the classical cognitive science goal of "Systematicity" (Fodor & Pylyshyn, 1988).

The "Broken" Combinatorial Space

The visual world is a product of concepts: objects, colors, positions, and textures. However, the training data for models like CLIP is a drop in the bucket of the combinatorial ocean.

The authors identify a critical gap: Why do models struggle with rare combinations? They argue that for a model to generalize, it must satisfy three Desiderata:

1. Divisibility: The readout must be able to "see" every part of the input.
2. Transferability: Classifiers trained on a subset must work on the full grid.
3. Stability: Retraining the model on different data shouldn't flip its "intuition" about what a concept (like "blue") means.

Methodology: The Geometry of Necessity

By analyzing Gradient Descent under Cross-Entropy (GD+CE), the authors prove that these desiderata force the representation space $Z$ to adopt a specific structure.

1. Linear Factorization

Each embedding $z$ must be a sum of independent concept vectors: $z_c={\sum }_{i=1}^k{u}_{i,c_i}$ This means the model treats an image as a "bag of concepts" where the final representation is simply the sum of its parts.

2. Cross-Concept Orthogonality

To prevent interference (e.g., the color "red" bleeding into the shape "square"), the directions of different concepts must be orthogonal.

Figure 1: Instantiating the framework within CLIP-like models, where text-derived probes act as linear classifiers on the image embedding space.

Experiments: Measuring the "Linearity Gap"

The authors surveyed modern SOTA models including CLIP, SigLIP, and DINOv3. Using a whitened $R^2$ metric, they measured how much of a model's embedding space is actually linear.

• The Linearity Correlation: There is a clear, positive correlation between how "linear" a model's representations are and its ability to generalize to unseen combinations.
• The $d\ge k$ Bottleneck: They proved that to distinguish $k$ different concepts, you need at least $k$ dimensions. In practice, models like SigLIP require even more ($\approx 2k$) to handle the competition between concepts.

Figure 2: Empirical evidence showing that as the Projected $R^2$ (linearity score) increases, the model's accuracy on held-out compositional combinations improves across all datasets (dSprites, MPI3D, PUG-Animal).

Deep Insight: Packing and Convergence

One of the most profound takeaways is the Factor-Rank Analysis. How do models fit hundreds of concepts into a fixed-size vector?

• Ordinal concepts (like size or position) occupy low-rank, almost 1D paths.
• Discrete concepts (like object identity) take up higher-dimensional "subspaces."

This suggests that as we scale models, they are not just getting "smarter"—they are converging toward a mathematically optimal, "Platonic" geometric arrangement that maximizes concept separation.

Conclusion & Limitations

Summary: This paper provides the "Ground Truth" for what efficient representation looks like. If you want a model to be compositionally robust, you must aim for linear, orthogonal factors.

Limitations: The theory assumes a linear readout. While many foundation models use linear probes/zero-shot heads, the role of non-linear "bottlenecks" in the backbone architecture remains a frontier. Furthermore, the "Stability" desideratum is a "worst-case" requirement that might be too strict for noisy, real-world web data.

Future Outlook: We can now use the "Projected $R^2$" as a diagnostic tool. Instead of running expensive benchmarks, we can simply measure the geometric linearity of an embedding space to predict its generalization potential.