Can pairwise convolutional networks infer cognitive traits from small neuroimaging samples?

Ranking pairs turns 108 people into 4,950 comparisons

The problem the paper targets is sample size. Conventional brain-behavior analyses relate a predefined structural measure, such as regional volume or cortical thickness, to a behavioral score through a linear model. With roughly a hundred participants, those models explain modest variance and are sensitive to noisy or discretized scores. Deep regression usually needs thousands of subjects to exploit nonlinear structure.

The authors change the prediction target. Instead of asking the network to output a person's absolute score, each training example contains two brain scans and a binary label saying which participant performed better. A dataset of N subjects yields N(N−1)/2 unique unordered pairs, so the 108-participant sample produced 4,950 pairs before augmentation such as left-right flips. The paper describes this as a quadratic expansion of the effective training signal, and it is the statistical reason the approach can be trained from scratch on so few people.

A shared slice encoder feeds a two-scan ranking network

The architecture has two stages. A slicing network takes a 3D T1-weighted volume, decomposes it into 2D slices, and passes them through residual blocks to produce a subject-level feature map. Both participants in a pair go through this encoder with shared weights, so the same anatomical representation is applied to each scan.

The two feature maps are then stacked along the channel dimension and passed to a ranking network, also built from residual blocks, whose fully connected head outputs the probability that one participant outperforms the other. Training minimizes binary cross-entropy between that predicted probability and the observed pairwise label. In the reported configuration the slicing network contains five residual blocks and the ranking network two, and roughly 10,000 ranking networks were trained across all region-elimination rounds in the 96×96×96 resolution experiment.

Maximum uncertainty marks the predicted continuous score

A ranking model alone does not give a continuous score, so the paper adds a reference-based inference step. Each test participant is compared against labeled reference participants spanning the observed score range, and the network returns the probability that the test participant scores higher than each reference. Because several references can share a score, those probabilities are averaged at each observed score.

The predicted score is read at the point of maximum decision entropy, where the model's probability of the test participant outperforming the reference is near 0.5. At that point the two people are judged most similar, so the reference score becomes the estimate. This is how a network trained without any regression loss still produces regression-like outputs; the paper illustrates the procedure for one participant on the mental rotation task.

Pairwise errors are lower than pointwise CNN errors on all four tasks

The comparison baseline was deliberately close in design: the same slice encoder, the same 192×192×192 MRI input, the same 2D slice pipeline and augmentation, but a single scan per example and a linear output trained with mean squared error. The pointwise baseline's mental rotation predictions collapsed into a narrow band near the training mean, with a mean absolute error of 0.924.

The pairwise model's continuous estimates tracked the identity line more closely. Test mean absolute errors were 0.647 for mental rotation (train 0.348), 2.625 for paper folding (train 0.245), 0.065 for immersive viewpoint transformation (train 0.069), and 3.125 for the road map task (train 1.071). The corresponding pointwise test errors were 2.893, 0.500, and 4.212 for paper folding, viewpoint transformation, and road map. The gap is largest where the pointwise model failed most, and the viewpoint transformation task is the exception in scale: both models have small errors there because that score is expressed on a different scale.

Removing one-third of regions per round localizes predictive cortex

To ask which anatomy the model actually uses, the authors ran iterative backward elimination. In each round, regions of interest defined by the FreeSurfer aparc+aseg parcellation, which includes the 68-region Desikan-Killiany cortical atlas plus subcortical structures, were ranked by their impact on model accuracy. The bottom third were discarded and the remaining regions carried forward, with the model retrained until the ranking stabilized. Each round held out 20% of subjects for evaluation, and early stopping on validation accuracy guarded against overfitting.

The resulting profiles included regions common to all four tasks and regions specific to some. Superior parietal cortex emerged for the road map and viewpoint transformation tasks, inferior parietal cortex for the road map task, and the precuneus primarily for the two object-based transformation tasks, mental rotation and paper folding. The viewpoint transformation task also showed postcentral involvement. Frontal regions appeared across tasks, which the authors read as distributed parietal, frontal, occipital, and sensorimotor contributions rather than a single locus. These are model-derived importance profiles, not direct measures of brain-behavior coupling.

Small samples and task-specific score scales bound the claims

The headline sample is 108 analyzed participants after four of 112 were excluded for low-quality scans or motion. Behavioral data were missing for some, so training used 106 for mental rotation, 107 for viewpoint transformation, 105 for road map, and all 108 for paper folding. The authors report that performance stayed consistent with as few as 50 training and 50 test subjects, which they treat as evidence the model was not memorizing individual anatomy.

Two cautions follow from the reported numbers. First, the four tasks are scored on different scales, so mean absolute errors cannot be compared across tasks as if they were the same unit; the viewpoint transformation error of 0.065 is small partly because its outcome is a position error on its own scale. Second, the anatomical maps come from removing atlas regions and watching accuracy change, which identifies regions the model relies on rather than proving that those regions cause individual differences. The paper's own framing is predictive and interpretable, not causal.

The priority is noisy labels, not bigger models

The paper's distinctive claim is not that a larger network would do better, but that changing the supervision target is what rescues small samples. The authors argue the comparative signal is less affected by score noise, discretization, and subject-specific measurement variability than an absolute target, and they note a separate result in which the same principle let lightweight CNNs trained from scratch reach competitive age estimation on UTKFace with far fewer images than usual.

That sets the practical priority for readers evaluating this method: it is most worth trying where absolute labels are scarce or noisy and where a single routine T1-weighted MRI is already available. The paper sketches a screening use for neurodegenerative disease, but that remains a proposal, not a tested result here. The demonstrated evidence is four spatial tasks in 108 young adults, with a road map replication in an independent dataset that had a training error of 1.862 and a test error of 4.625, larger than in the original sample but still below the baseline. The decision-relevant comparison is therefore pairwise ranking versus pointwise regression on the same small structural MRI sample, not pairwise ranking versus large-scale deep learning.

Sources used in this answer