Trustworthy ML in Healthcare: How to Value 100,000 Medical Images with Scalable Shapley Values

TL;DR

In the quest for trustworthy AI in medicine, we often ask: "Which data points actually matter?" This paper tackles the scalability wall of data valuation. By shifting from exhaustive model retraining to a KNN-Shapley approximation, the researchers successfully ranked the importance of over 100,000 chest X-rays. This paves the way for automated label cleaning and fair data marketplaces in healthcare.

Academic Positioning: This work is a crucial experimental validation of scalable data valuation algorithms (specifically KNN-Shapley) on a large-scale, real-world clinical dataset (CheXpert), moving beyond the toy examples often seen in algorithmic theory papers.

The Problem: The "Black Box" of Data Contribution

Machine Learning (ML) thrives on data, but not all data is created equal. In medical imaging, a dataset might contain:

1. High-quality gold standards: Expert-labeled, clear diagnostic features.
2. Redundant samples: Thousands of nearly identical "normal" scans.
3. Noisy/Mislabeled data: Scans where the "uncertainty" label (NaN or U) masks an underlying condition.

Existing methods like Leave-One-Out (LOO) require retraining the model for every single data point removed—a nightmare for deep CNNs. TMC-Shapley is better but still relies on frequent retraining. These methods hit a "scalability wall" at around 1,000 samples.

The Insight: Leveraging KNN and Deep Features

The authors propose a shift in perspective. Instead of valuing data through the lens of a heavy DenseNet-121, they use the DenseNet as a feature extractor and perform valuation using a k-nearest neighbor (KNN) proxy.

Why does this work?

The KNN-Shapley value satisfies the core requirements of "fair" valuation (Group Rationality, Fairness, and Additivity) but allows for recursive calculation. The complexity becomes independent of the neural network's depth and instead depends on the dimension of the embeddings.

Figure 1: Comparison of valuation workflows. Note how KNN-Shapley bypasses the retraining loop.

Methodology: The Workflow

1. Pre-training: Train a DenseNet-121 on the CheXpert dataset to detect Pleural Effusion.
2. Feature Extraction: Extract "Deep Features" from the final dense blocks.
3. KNN Approximation: Build a KNN classifier where $k$ is tuned to match the CNN’s performance.
4. Recursive Calculation: Use the linearized Shapley formula to assign values to all 100,000 training instances.

Table 1: The modified DenseNet-121 used as the backbone for feature extraction.

Experiments: Proving the Value

The authors conducted two high-impact experiments to validate their scores:

1. The "Removal" Test

If a data point is truly "high value," removing it should tank the model's performance. The results were stark: removing high-value points caused a massive drop in AUC, while removing low-value points (likely redundant or noisy) had almost no negative impact—sometimes even improving the model slightly.

2. Noisy Label Detection

In a simulated "noisy" environment where 10% of labels were flipped, the KNN-Shapley value acted as a "garbage detector." 80% of the errors were found within the bottom 30% of the ranked data. This allows radiologists to focus their limited time on auditing only the most suspicious samples.

Figure 2: The efficiency of identifying mislabeled data using the bottom-ranked Shapley values.

Critical Insight & Conclusion

The true value of this paper lies in its industry applicability. By proving that KNN-Shapley scales to 100,000 images, the authors provide the technical "receipt" needed for:

• Data Marketplaces: Paying hospitals for their data based on its actual marginal utility.
• Privacy-Preserving Summarization: Reducing datasets by removing low-value, high-sensitivity outliers without losing accuracy.

Limitations: The study focuses on a single condition (Pleural Effusion). Future research must investigate if these values remain stable in multi-label settings where a single scan might be "high value" for detecting Pneumonia but "low value" for Fractures.

Conclusion: Data valuation is no longer an academic exercise for small datasets. With KNN-Shapley, we have the tools to audit massive clinical repositories, ensuring the "data foundation" of healthcare AI is both accurate and fair.