Readout blindness names a gap between encoder and score
CLIP-like vision-language models match images and captions by scoring separately encoded representations, and their vision encoders also feed multimodal large language models. The paper focuses on the score-based readout, which supports ranking and thresholding and is therefore used for decisions. Those scores remain surprisingly weak on directed spatial relations such as whether one object is left of another, often near chance even when the relevant objects are recognized.
The paper calls this failure readout blindness: the gap between what the encoder retains and what the score reveals. Its central claim is that directional failure can arise from the scoring rule itself, even when the information is present in the encoder. That reframes the problem from learning a better representation to designing a better readout, and it is the question the rest of the paper tests.
Symmetric scoring rules cancel direction by construction
The theoretical part of the analysis asks when a scoring rule cannot separate directions. The paper's answer is that when scoring rules treat the subject and object symmetrically, direction cancels regardless of encoder training. In other words, a rule that does not distinguish which object is the subject and which is the object cannot express which one is left of the other.
This is a claim about the readout, not about the encoder's capacity. It also explains why improving accuracy through fine-tuning or extra supervision does not settle the original question: gains may come from newly learned spatial cues or from text and world priors rather than from the encoder exposing direction. The paper therefore keeps the encoder fixed and examines what can be recovered from its features.
ADR scores direction as signed displacement between centroids
Antisymmetric Displacement Readout (ADR) aligns caption words with image patches in the frozen features and scores each relation by the signed displacement between matched object centroids. The alignment is explicit: the method computes a correspondence between text tokens and image regions and exposes it, so the alignment maps and centroids help explain each decision.
Crucially, ADR requires neither tuning nor additional learned parameters. It succeeds without additional training, which the paper treats as evidence that directional information remains in the frozen encoder. The paper reports two variants: ADR with a Sinkhorn-based coupling that retains excess mass, and ADR-pi-d, a closed-form softmax over patches computed independently for each word. The default variant uses the excess coupling, and the implementation floors column mass at 10^-30; no empty columns were observed in published runs.
Prior deflation separates real alignment from text and world priors
The paper notes that text and world priors can inflate accuracy, which makes accuracy alone unreliable for evaluating alignment. A readout can score well without using the matched image at all. To address this, the authors introduce prior deflation: each caption is paired with an unrelated image from the same benchmark and scored again, a control they call the derangement null.
Prior deflation subtracts null accuracy from raw accuracy to obtain grounded gain, which measures the benefit of the original image-caption pairing after accounting for prior effects. The paper also constructs direction-balanced subsets with near-chance null accuracies for fairer comparisons. These subsets, prior deflation, reproducible derangement controls and shared scoring rules form a reusable evaluation kit.
ADR gains grounded accuracy across encoder families
Across encoder families, ADR substantially improves over deployed scores, which remain near chance on most direction-balanced sets even for fine-tuned encoders. On the VG-Relation left/right subset (n=3918), ADR-pi-d reaches 91.04% raw accuracy against a 49.62% null, a grounded gain of +41.4 percentage points, while the deployed cosine score sits at 50.33% raw with a 48.72% null and a +1.6-point gain.
The pattern holds on other sets. On COCO-spatial (n=404), ADR-pi-d reaches 94.80% raw against a 48.02% null (+46.8 points), and on What'sUp-B (n=354) it reaches 52.70% raw against a 25.00% null (+27.7 points). On the VSR horizontal subset (n=1201), ADR-pi-d reaches 100.00% raw against a 51.42% null (+48.6 points). The paper reports these as grounded gains after prior deflation, not as raw accuracy alone.
ADR beats MLLM likelihood readouts at a fraction of the cost
Compared with more complex readouts, ADR outperforms the evaluated MLLM likelihood readouts and is competitive with their chat inference at a small fraction of the computation. In the AUROC analysis under the derangement null, ADR-pi-d reaches 0.880 on VG-Rel left/right and 0.899 on COCO, while the Qwen3.6-27B likelihood readout reaches 0.717 and 0.726 respectively; the deployed cosine score stays near 0.5.
At 30% coverage, ADR-pi-d reaches 0.999 accuracy on VG-Rel left/right and 1.000 on COCO, against 0.916 and 0.967 for the Qwen likelihood readout and 0.495 and 0.446 for deployed cosine. The cost comparison is also stark: ADR uses the same 0.15B parameters as deployed CLIP, about 3.3x its FLOPs and 2.2x its time, whereas Qwen3.6-27B chat uses 182.4x the parameters, roughly 550.7x the FLOPs and 65.9x the time.
Depth proxies and reference frames bound where ADR transfers
For front/behind relations, ADR uses the image vertical axis as a depth cue rather than estimating three-dimensional depth, so correct object localization alone does not guarantee the correct 3D relation. The paper states that this axis was chosen after exploratory VG-Relation analysis and fixed before evaluation on disjoint VG-Relation images and What'sUp-B, and that the absence of training refers to encoder weights and learned readout parameters.
Reference-frame annotations in VSR explain part of the remaining error. On true statements, ADR agrees with relations defined from the viewer's perspective at a rate of 0.9290, but with those defined from the object's perspective at only 0.2239. Because ADR measures displacement in image coordinates, it follows the viewer's perspective, and this mismatch with object-centered annotations accounts for some of its errors on VSR. Temperature sensitivity also differs by variant: across 0.02 to 0.2, left/right accuracy varies by 0.1 points for ADR-pi and 1.6 for ADR-pi-d, while on What'sUp-A the ranges are 2.9 and 28.7 points.
The readout, not the encoder, is the place to look first
The paper's most decision-relevant claim is a priority ordering: before fine-tuning an encoder or adding spatial supervision, check whether the scoring rule can express direction at all. A symmetric rule cancels direction regardless of encoder training, so improvements from retraining may reflect new cues or priors rather than recovered direction. ADR shows that directional information can be recovered from frozen features by an appropriate readout, without training or learned parameters.
The practical threshold the paper supplies is grounded gain, not raw accuracy. On direction-balanced subsets with near-chance nulls, ADR's grounded gains are large and consistent across encoder families, while deployed scores stay near chance even for fine-tuned encoders. The comparison that matters most is therefore between readouts evaluated under prior deflation on the same items, derangement assignments and scoring rules, with the viewer-versus-object reference frame kept in view for depth and VSR-style relations.
Scope stays with noun-pair relations and frozen encoders
The evidence covers directed spatial relations between noun pairs, evaluated on direction-balanced subsets drawn from ARO, What'sUp, VSR and VALSE, with encoder families including CLIP, MetaCLIP, SigLIP2, BLIP, NegCLIP, CLoVe and CE-CLIP. The paper's own discussion lists broader multimodal alignment tasks, more complex relations beyond noun pairs, and extensions to a wider range of VLM architectures as future work.
Two boundaries qualify the results. First, the depth relation is a vertical-axis proxy, not a 3D estimate, so localization success does not imply correct front/behind judgment. Second, the evaluation kit's direction-balanced subsets are selected for near-chance null accuracies, which is what makes grounded gain interpretable but also means the reported gains are specific to those controlled sets rather than to arbitrary benchmarks.
Sources used in this answer
Readout Blindness: VLM Scores Miss the Spatial Direction Their Frozen Encoders Retain
Symmetric scoring rules cancel spatial direction regardless of encoder training, and the training-free Antisymmetric Displacement Readout recovers that direction from frozen features, with large grounded gains over deployed scores across encoder families.