Do Foundation Models Know Geometry? Bridging the 3.3x Text Bottleneck
Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement
Summary
Problem
Method
Results
Takeaways
Abstract
This paper investigates whether Vision-Language Models (VLMs) internally represent continuous physical geometry despite their poor text-based performance. By applying 6,000-parameter linear probes to frozen features across 14 foundation models, the author extracts geometric measurements (like hand joint angles) at high accuracy (6.1° MAE), significantly outperforming the models' own text outputs (20.0° MAE).
## TL;DR
While modern AI models like Qwen or Gemma might fail to tell you the exact angle of a human finger in text, their internal "frozen" neurons actually know the answer with surprising precision. This paper demonstrates that a simple linear probe can extract geometric data at **6.1° MAE**, while the text output remains stuck at **20.0°**. The secret? It's not the architecture (ViT vs. CNN), but the **training objective** that forces models to learn the "Platonic" laws of geometry.
---
## The Problem: The "Text Bottleneck"
We often ask Vision-Language Models (VLMs) to describe the world, but when it comes to physical measurements—like the tilt of a head or the flexion of a knuckle—they hallucinate or provide coarse, imprecise estimates.
The central question of this research is: **Is the model "blind" to geometry, or is it just "speechless"?**
By probing the frozen features of 14 different foundation models, the author discovered that the visual representations are far more geometrically aware than the text they produce. There is a "bottleneck" where the autoregressive decoder discards fine-grained spatial details in favor of linguistic fluency.
---
## Methodology: Probing the Latent Space
The author utilized **Reduced-Rank Ridge Regression (RRR)** to map high-dimensional hidden activations to continuous geometric targets.
### Key Approach:
1. **Frozen Backbones**: 14 models including DINOv3, SigLIP 2, and Qwen2.5-VL were used without updating their weights.
2. **Linear Probes**: A tiny 6,000-parameter layer was trained to "read" the geometry from the frozen features.
3. **Cross-Domain Validation**: Tests covered hand pose (FreiHAND), head pose (BIWI), object pose (YCB-Video), and gaze direction.
*Figure 1: The gap between what a model "sees" (Probe) and what it "says" (Text).*
---
## Core Insights
### 1. Training Objective Over Architecture
The most striking finding is that whether you use a Vision Transformer (ViT) or a Convolutional Neural Network (CNN) matters less than *how* the model was trained.
* **Self-supervised (DINO)** and **Contrastive (CLIP/SigLIP)** models far outperformed supervised-only models.
* Models with wildly different internal "languages" (low CKA similarity) arrived at the same level of geometric accuracy. This is a phenomenon the author calls **Functional Convergence without Representational Convergence**.
### 2. The Proximal-Distal Gradient
Geometry is not distributed equally. The probes found it much easier to predict the base of a finger (MCP joints) than the tips (DIP joints). This suggests that models capture the "structural trunk" of pose more strongly than the "articulated leaves."
### 3. LoRA as a "Routing" Solution
If the geometry is there but the text is wrong, can we fix it? By applying **LoRA (Low-Rank Adaptation)** with only 2,000 images, the author was able to "route" the internal geometric knowledge into the text pathway, bringing the text error down from 20.0° to 6.5°.
*Figure 2: Layer Trajectory showing how geometric signal builds in vision encoders but often declines in LLM decoders.*
---
## Experimental Results: The 0.55 R² Ceiling
The research identified a statistical "equivalence cluster." Five distinct models—SigLIP 2, DINOv3, CLIP, SigLIP, and InternViT—all hit a performance ceiling of roughly **R² ≈ 0.55**.
| Model | Training Objective | MAE (Lower is better) |
| :--- | :--- | :--- |
| **SigLIP 2 (Frozen)** | Hybrid VL | **6.14°** |
| **MediaPipe (Task-Specific)**| Hand-Specific | 16.3° |
| **Qwen-3B (Text output)** | Generative | 20.0° |
This suggests that while foundation models aren't "perfect" at geometry, they are significantly more capable than task-specific tools like MediaPipe when measured against high-fidelity ground truth.
---
## Critical Analysis & Conclusion
### Limitations
The "Platonic" convergence stops at a certain point—about 45% of geometric variance remains unexplained by linear probes. This indicates that while the "logic" of geometry is captured, the "precision" might require non-linear modules or higher-resolution inputs.
### Future Outlook: Modular Geometric Sensing
This paper proposes a shift in how we build spatial AI. Instead of training massive new models for every physical task, we can use a **single frozen backbone** as a "universal sensor."
By adding a few thousand parameters (a "probe") for each specific task (head pose, camera focal length, joint angles), we can transform a general-purpose VLM into a multi-task physical measurement tool at almost zero computational cost.
**The takeaway for practitioners**: Your VLM already knows the physics of the scene; you just need to build a better bridge to extract it.