Can gated progressive fusion improve polyp re-identification across colonoscopy images?

Matching the same polyp across separate examinations

Polyp re-identification (ReID) asks a model to retrieve a specific polyp from a large gallery of colonoscopy images taken from different viewpoints and with different cameras. The paper frames this as a retrieval problem rather than detection or segmentation: the model must decide which gallery images show the same lesion, even though background, viewpoint and illumination change between captures.

That framing matters for reading the results. Detection and segmentation methods, which the paper says dominate existing colonoscopy computer-aided diagnosis work, address a single examination. ReID instead has to link the same polyp across examinations, which the paper describes as complicated by viewpoint changes, illumination differences, bowel deformation and tissue evolution. The benchmark used here is the polypus dataset, and the reported comparisons are against ReID models rather than against detection or segmentation systems.

Coarse high-level features limit small-polyp matching

The paper's stated motivation is a granularity problem. High-level features are coarse, and that coarseness limits performance on small polyps, where fine-grained details are what make an accurate match possible. GPF-Net is proposed as a way to keep those fine-grained details available while still using higher-level semantic information.

The mechanism the authors describe is a coarse-to-fine alignment. The fusion begins by separating low-level discriminative features such as texture and spectral signatures, then hierarchically connects them with high-level semantic constructs including object categories and relational contexts. The authors argue this preserves fine-grained detail and reduces cross-modal interference, which they present as a correction to single-stage end-to-end fusion. This is the paper's own explanation of why the architecture should help; the evidence offered for it is the benchmark comparison and the attention-map visualization.

Two encoders feed a four-layer gated fusion stack

GPF-Net has two parts: a visual-textual feature extraction module and a two-stage feature fusion module. A pre-trained ResNet-50 encodes a 224×224×3 polyp image into a 1×2048-dimensional vector, which is reduced to a 512-dimensional image feature. A pre-trained ALBERT model encodes a colonoscopy report description of n tokens into an n×768-dimensional text feature.

The image and text features are concatenated and given positional encoding, then passed into a 4-layer gated progressive fusion network. The fused feature that comes out is used to compute the training loss. Training combines identity loss, which strengthens the model's ability to classify polyp categories, with triplet loss, which pulls similar polyp images closer and pushes dissimilar ones apart. The paper states that this multi-loss choice follows earlier findings that multiple losses support robust and generalizable ReID models.

A sigmoid gate reweights image and text at each layer

The gating step is concrete. The image feature passes through a fully connected layer with a learnable matrix, and a sigmoid activation turns that output into a gating weight matrix. That matrix sets the proportions of the image and text modalities in the fused feature: the gated result is the gate times the text feature plus one minus the gate times the image feature.

Unlike traditional gated fusion that uses the weighted vector directly as the fused feature, GPF-Net passes the gated result through layer normalization together with the image feature, then uses multi-head attention to map it into a higher-dimensional space. The paper describes this as enabling deeper semantic extraction from the fused vector. The architecture uses 8-head self-attention inside the 4-layer gated progressive fusion network and 4-head self-attention in the final 4-layer Transformer encoder, a choice the authors present as balancing feature extraction against parameter and compute cost.

Multimodal fusion beats unimodal ReID baselines

The headline result is a comparison, not an absolute score. On the polypus dataset, the authors report that GPF-Net achieves state-of-the-art performance and outperforms existing methods by significant margins in mAP and Rank-1 accuracy. The abstract states the advantage of the multimodal setting over state-of-the-art unimodal ReID models, particularly when combined with the fusion strategy tailored for general-purpose scenarios.

Two cautions belong next to that result. First, mAP and Rank-1 are retrieval metrics, and the paper reports them as margins over existing methods rather than as a single deployment threshold; a reader should not convert them into a clinical accuracy figure. Second, the comparison is against unimodal ReID models on one benchmark, so the finding supports the claim that adding report text helps on this dataset, not that it helps equally in every hospital or camera setup.

Attention maps show broader context with text fusion

Beyond the retrieval metrics, the paper visualizes attention maps under three input settings: original images, a CNN-based training method without text information, and the gated progressive fusion strategy. The authors state that their model learns global context and meaningful visual features, improving multimodal understanding and enabling automatic discovery of deeper, more representative image features.

This visualization is supporting evidence for the authors' interpretation, not an independent measurement of matching accuracy. It is consistent with the paper's argument that text supplies complementary semantic information beyond visual appearance, but the quantitative claim still rests on the mAP and Rank-1 comparison on the polypus benchmark.

The gain is demonstrated on one benchmark

The evidence boundary is the polypus dataset and the paper's chosen baselines. The paper motivates the work with a practical problem, that manually labeling large numbers of pairwise polyp area data is time-consuming and labor-intensive when deploying a polyp ReID system to new hospital scenarios, but it does not report a cross-hospital deployment study. The reported advantage therefore applies to the benchmark setting the authors evaluated.

The paper also positions GPF-Net against two named limitations of prior multimodal polyp ReID work: shallow fusion strategies that may not fully exploit complementary semantic information, and increasingly complex architectures that add computational overhead and parameter redundancy. GPF-Net's answer is dynamic modality weighting plus progressive cross-modal interaction, with the 8-head and 4-head attention split intended to keep the model lightweight. Whether that trade-off holds outside the reported experiments is not tested here.

Text fusion is the variable worth watching

The paper's most decision-relevant comparison is not GPF-Net against a generic baseline but the multimodal setting against unimodal ReID models on the same benchmark. That comparison isolates what the colonoscopy report text adds, and the authors report that the gain is largest when the gated progressive fusion strategy is used. A reader weighing multimodal polyp ReID should therefore treat the availability and quality of report text as a first-order condition, because the architecture's advantage is defined relative to models that lack that input.

The second condition is granularity. The paper's stated failure mode for prior work is coarse high-level features on small polyps, and its proposed fix is layer-wise refinement across four gated fusion layers. That makes small-polyp matching the scenario where the design is meant to matter most, and it is the specific claim a follow-up evaluation should test rather than the aggregate benchmark margin alone.

Sources used in this answer