[CVPR 2025] A Dataset is Worth 1 MB: Eliminating Pixel Transmission via PLADA

TL;DR

PLADA (Pseudo-Labels as Data) is a radical rethink of dataset serving. Instead of sending raw images (GBs) or synthetic distilled images (MBs), it assumes the client already has a "dictionary" of images (ImageNet). The server simply sends a list of labels for a subset of these images. This reduces the cost of transferring a training set to less than 1 MB, making it feasible for extreme environments like deep-sea exploration or space missions.

Background: Why Pixels are the Wrong Primitive

Transmitting training data to remote agents (autonomous vehicles, medical devices, or underwater robots) is a bottleneck. Sending pre-trained models often fails because clients use different hardware or frameworks. However, sending raw data is energetically and temporally prohibitive in bandwidth-constrained environments (e.g., 500 bps satellite links).

While Dataset Distillation tries to help by creating small synthetic sets, it has three massive flaws:

1. Optimization Instability: Scaling to high-resolution (like ImageNet-21K) is a nightmare.
2. Pixel Overhead: Full-precision synthetic pixels are still relatively large.
3. Architecture Bias: Distilled images are often coupled to the architecture used during distillation.

Methodology: High-Level Intuition

The authors' insight is simple: Labels are dense; pixels are sparse. If we can find "proxy" images on the client side that look like the target task, we only need to tell the client what those images represent.

1. The Pipeline

1. Teacher Training: The server trains a high-quality model on the target task.
2. Pseudo-Labeling: The teacher labels a massive reference set (like ImageNet-21K) already stored on the client.
3. Pruning (The Filter): Not all ImageNet images are relevant. The server identifies which images "look like" the task using an Energy-based OOD score.
4. Safety-Net: To prevent "easy" classes from dominating, a quota-based system ensures rare classes are preserved.
5. Compression: The indices and labels are compressed using Zstd, shrinking the "dataset" to a tiny binary payload.

2. Pruning as a Denoising Mechanism

Surprisingly, training on 1% of the filtered reference dataset often works better than training on 100%. This is because high-energy (uncertain) images act as noise. By filtering for low-energy samples, the server selects "canonical" exemplars that structurally align with the target domain.

Experiments: Breaking the 1 MB Barrier

PLADA establishes a new Pareto frontier. In the sub-1 MB range, traditional methods like Coresets or Random Subsampling fail because they can only afford to send a handful of images, which isn't enough for a model to generalize.

Key Results:

• Fine-Grained Classification: On CUB-200 (birds), PLADA achieves 82.5% accuracy with just ~150 KB, while Dataset Distillation (DD) baselines struggle to reach 20% in similar regimes.
• Scale Matters: Using ImageNet-21K (14M images) as a reference is significantly better than ImageNet-1K, as the "dictionary" is more likely to contain near-neighbors for niche tasks.

The "Medical" Exception: A Surprising Discovery

When the target task is radically different from the reference set (e.g., Medical Blood Work vs. Natural Images), the authors found an Energy Paradox. Low-energy (high confidence) natural images were actually destructive because they represented "spuriously" similar textures. In these "Far-OOD" cases, selecting high-energy (uncertain) images worked better, presumably because these images contain the complex high-frequency patterns needed to simulate medical scans.

Critical Analysis & Conclusion

Takeaway: PLADA proves that for classification, we don't need to send the "visuals" if the client already has a diverse enough world model. A "Dataset" is really just a set of pointers to concepts.

Limitations:

• Storage: The client must store a massive (though unlabeled) reference set.
• Domain Gap: If the reference set is too narrow (e.g., only containing cars), it won't be able to serve a task about flowers.
• Task Scope: Currently limited to classification; extending this to generative tasks (where pixels must be new) remains the "holy grail."

PLADA represents a significant shift: moving from "transmitting data" to "communicating knowledge" through existing shared priors.