Why Contrastive Learning "Hiddenly" Creates Gaussian Representations

In the world of Self-Supervised Learning (SSL), the InfoNCE loss is king. It powers the most successful models we have today, from SimCLR and MoCo to the image-text giant CLIP. We’ve always known that InfoNCE pushes representations to be uniform on a hypersphere, but a new paper from Technion researchers reveals a much deeper probabilistic truth: InfoNCE induces a Gaussian structure in the representation space.

TL;DR

This research provides the first principled population-level explanation for why representations trained with contrastive objectives look like Gaussians in high dimensions. By combining the geometry of the hypersphere with the Maxwell-Poincaré spherical central limit theorem, the authors prove that high-dimensional projections of these features asymptotically approach a multivariate Gaussian distribution.

The "Uniformity" Missing Link

Usually, we describe contrastive learning as a balance between alignment (clumping positive pairs together) and uniformity (spreading different samples apart).

The authors argue that as the dimension $d$ grows, "spreading out" on a sphere actually means something very specific in Euclidean space. Through the lens of high-dimensional probability, a thin shell on a high-dimensional sphere effectively behaves like a Gaussian "cloud."

Methodology: The Two Routes to Gaussianity

The paper establishes Gaussianity through two distinct mathematical paths:

1. The Empirical Idealization Route

This route relies on two observations commonly seen in practice:

• Alignment Plateau: Positive pair similarity eventually hits a ceiling determined by how "strong" the data augmentations are.
• Thin-Shell Concentration: Over time, the norms (lengths) of feature vectors concentrate around a single value, effectively placing data on a "shell."

2. The Regularized Route

To prove this without relying purely on training dynamics, the authors analysis a regularized population objective. By adding a vanishing term that favors high entropy and low feature norms, they show the isotropic Gaussian solution is the unique global minimizer.

Figure 1: Conceptual illustration showing how contrastive objectives transform arbitrary data distributions into Gaussian representation clusters.

The HGR Alignment Bound

One of the most elegant parts of the paper is the use of Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. It provides a theoretical limit: you cannot align positive pairs better than the "information" shared between the original image and its augmented version. This ${\eta }_2$ parameter acts as a speed limit for the alignment part of InfoNCE.

Experimental Proof: Contrastive vs. Supervised

The authors didn't just stop at math. They tested linear encoders, MLPs, and ResNet-18s on synthetic and real-world datasets like CIFAR-10 and MS-COCO.

The results are striking. While supervised models (using Cross-Entropy) create messy, non-Gaussian feature spaces, InfoNCE models consistently produce features that pass standard normality tests (Anderson-Darling and D’Agostino-Pearson).

Figure 2: Analysis showing that while alignment saturates early, uniformity continues to improve with batch size, driving the features toward a Gaussian state.

Deep Insight: Why Does This Matter?

Is this just a theoretical curiosity? No. Understanding that representations are Gaussian has immediate practical benefits:

1. Out-of-Distribution (OOD) Detection: If features are Gaussian, we can use simple Mahalanobis distances or Likelihood scores to detect "strange" data with high precision.
2. Uncertainty Estimation: Closed-form solutions for entropy and KL-divergence become available, making probabilistic modeling much cheaper.
3. Principled Design: It justifies why "whitening" (decorrelating) features often helps—it essentially acts as a shortcut to the Gaussian ideal the loss is already trying to reach.

Conclusion

This work bridges the gap between the geometric intuition of "uniformity on the sphere" and the probabilistic reality of Gaussian embeddings. It confirms that the "implicit bias" of InfoNCE is to create the most statistically "neutral" and spread-out distribution possible: the Gaussian distribution.

Reference: Betser et al., "InfoNCE Induces Gaussian Distribution", Technion - Israel Institute of Technology.