How can brain MRI scans be retrieved quickly for whole-brain and region-specific queries?

NeuroCBIR retrieves similar scans, not diagnoses

Content-based image retrieval (CBIR) in neuroimaging aims to find brain scans that resemble a query scan based on intrinsic image features rather than metadata. NeuroCBIR is a framework for fast and flexible retrieval of both whole-brain and region-specific 3D T1w MRI scans. It extracts 103 cortical and subcortical regions, so a user can query either an entire brain or a single anatomical structure.

The system does not output a diagnosis. Its primary measured task is subject re-identification: given a query scan, does the system rank other scans from the same subject above scans from different subjects? That is a retrieval question, and it is evaluated separately from the downstream zero-shot tasks described later.

A VAE and contrastive learning build the embedding space

NeuroCBIR uses a two-stage pipeline. First, a variational autoencoder (VAE) learns latent representations of the input volumes. The authors trained separate VAE models for whole-brain and brain-region data, using an encoder-bottleneck-decoder structure with a latent representation of 8 channels. For whole-brain input the latent shape is [8,20,22,26]; for region input it is [8,8,8,8].

Second, those precomputed latent representations are passed to an encoder-projector model trained with contrastive learning. The contrastive objective pulls together intra-subject images and pushes apart inter-subject images, producing compact embeddings for similarity comparison. The paper describes this combined block as the Query-to-Embedding (Q2E) module. At inference, a query image is mapped to its embedding, compared against a reference embedding database using pairwise cosine similarity, and the top-k most similar images are returned.

The authors motivate the latent-representation approach partly by memory cost: training the VAE with a batch size of one whole-brain MRI required about 63 GB of GPU memory in their experiments. Once trained, the encoder alone is used for embedding extraction.

Re-identification reaches mAP@5 of at least 98.4%

The headline retrieval result is high across both whole-brain and brain-region levels: mean average precision across the top-5 retrieved images (mAP@5) is at least 98.4%. The paper reports robust generalization across datasets and acquisition conditions.

The evaluation used more than 26,000 MRI scans from five datasets: ADNI, OASIS3, AIBL, MIRIAD, and SLIM. ADNI and OASIS3 were used for training, validation, and testing, while AIBL, MIRIAD, and SLIM were reserved exclusively for external testing. The datasets span a range of acquisition protocols, scanner manufacturers, and field strengths, including 1.5T and 3.0T.

The paper also reports a Spearman correlation between the embedding space and MS-SSIM, which the authors present as evidence that the learned embeddings preserve meaningful perceptual and semantic similarities between inter-subject scans. That correlation supports the interpretation of the embedding space; it is not the same measure as mAP@5.

Zero-shot age and pathology results show encoded structure

NeuroCBIR is not trained for age prediction or pathology stratification. The authors nevertheless evaluated the embeddings on these tasks without task-specific supervision, to test whether the representations encode clinically meaningful information.

For zero-shot age prediction, the method achieves a mean absolute error of about 4.3 years and a Pearson correlation of about 0.5 across the evaluated brain regions. On the external AIBL cohort, NeuroCBIR achieved an MAE of 4.5 years, compared with 3.8 years reported by the supervised brain age estimation model of Rajabli et al. (2025). The authors note that their method is not tailored for age prediction and describe it as complementary to supervised brain age regression, potentially adding interpretability by inspecting age differences between the query and retrieved scans.

For zero-shot pathology prediction, whole-brain embeddings achieve a balanced accuracy of 56.3% and an MF1 score of 53.1% across CN, MCI, and AD. On the external AIBL cohort, NeuroCBIR achieved balanced accuracies of 75.2% using whole-brain embeddings and 81.9% using the left amygdala for the CN/AD classification task, without retraining or domain adaptation. The authors compare the 81.9% figure to the 84.0% reported by the supervised classifier of Wen et al. (2020).

The paper also reports that most misclassifications between CN and AD occur toward the MCI class, which the authors interpret as reflecting MCI's intermediate clinical status. This is an author interpretation of the confusion pattern, not a separate clinical validation.

Region-level retrieval varies by anatomical structure

Because NeuroCBIR supports region-specific queries, the paper reports retrieval and classification performance for individual brain regions. In the region-based approach, the contrastive loss is computed independently per brain region, and at each training iteration 10 regions are randomly selected from the 103 regions within the batch.

For the region-level zero-shot classification results reported in the paper, the left amygdala reaches a balanced accuracy of 81.9% and an MF1 of 76.3% for the CN/AD task on the AIBL query set, the left hippocampus reaches 79.5% balanced accuracy and 73.1% MF1, the left thalamus reaches 72.6% and 63.1%, and the left lateral ventricle reaches 72.3% and 64.3%. These are region-specific numbers and should not be read as whole-brain performance.

The authors frame region-based retrieval as useful when regional effects dominate over global structure, enabling retrieval focused on anatomically localized patterns for task-specific stratification.

Embedding extraction is the slow step, search is near-instant

The paper separates two costs. Embedding extraction on a 4-core CPU required about 18.7 seconds per scan. Similarity search was effectively instantaneous, reported as less than 0.01 seconds. The practical implication is that the expensive step happens once per scan when building or extending the reference database, while querying that database is fast.

NeuroCBIR is publicly available for brain MRI with more than 26,000 precomputed T1w MRI embeddings. The authors state that new scans can be added directly to the database without retraining the model, so coverage can grow as additional data becomes available. They also note that the framework uses only image similarity during training, which is intended to ease integration into systems such as TheHiveDB, where retrieval results could be filtered by additional criteria such as diagnosis, age, sex, biomarkers, memory tests, or scanner.

Preprocessing and sequence scope bound the current evidence

The evaluation used skull-stripped, bias field-corrected T1w images processed according to the FreeSurfer pipeline, affinely registered to FreeSurfer Talairach space, rescaled to [0,1], and cropped to [160, 176, 208]. Brain region masks came from FreeSurfer v5.3.0 in TheHiveDB for ADNI and OASIS3, and from FastSurfer v2.4.2 for AIBL, MIRIAD, and SLIM.

The authors state that although the system was evaluated on data preprocessed with pipelines slightly different from the training data, it remains unclear how robust the model is to substantially different preprocessing strategies. They also note that the current work focuses on T1w MRI and that future work should extend the framework to other clinically relevant sequences such as FLAIR.

The paper describes the study as primarily focused on development and technical validation of NeuroCBIR. The zero-shot age and pathology evaluations are exploratory, and the authors call for a more comprehensive investigation of these and additional clinical applications. The reported correlation among image similarity, age, and condition is described as worth further study, not as an established causal relationship.

The region choice can matter more than the global score

The most decision-relevant comparison in this paper is not whole-brain versus region-level retrieval in general, but which region is queried. On the external AIBL CN/AD task, whole-brain embeddings reached 75.2% balanced accuracy, while the left amygdala reached 81.9% without retraining or domain adaptation. That gap shows that a region-specific query can carry more discriminative signal than the whole-brain embedding for a particular downstream question.

At the same time, the paper's strongest and most consistent result remains re-identification, where mAP@5 is at least 98.4% across whole-brain and region levels. The zero-shot age and pathology numbers are lower and explicitly exploratory. For a reader deciding whether to use NeuroCBIR, the practical priority is therefore to treat retrieval quality as the validated capability, treat zero-shot clinical stratification as a promising but unvalidated extension, and choose the query region deliberately rather than assuming whole-brain is always the best default.

Sources used in this answer