[CVPR 2026] Bootleg: "Using the Whole Buffalo" via Hidden-Layer Self-Distillation
Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
The paper introduces Bootleg, a single-view self-supervised learning (SSL) method that utilizes "hidden-self-distillation." By predicting latent representations from multiple hierarchical hidden layers of an EMA teacher network rather than just the final layer, Bootleg achieves SOTA performance among non-contrastive methods, notably outperforming I-JEPA by +10% on ImageNet-1K classification.
TL;DR
Bootleg is a breakthrough in Self-Supervised Learning (SSL) that solves the instability of predictive models like I-JEPA and the semantic shallowness of generative models like MAE. By predicting representations from multiple hidden layers of a teacher network simultaneously, Bootleg learns a rich hierarchy of features. It achieves a massive +10% accuracy boost over I-JEPA on ImageNet-1K while remaining computationally accessible (single-GPU friendly).
The Missing Link in Representation Learning
In the current SSL landscape, we have two dominant but flawed paradigms:
- Generative (e.g., MAE): These "fill-in-the-pixel" models are grounded in reality but often fail to learn high-level concepts because pixels are too low-level.
- Predictive (e.g., I-JEPA): These predict the final "thought" (embedding) of a teacher network. While more semantic, they are notoriously unstable because the "target" keeps moving, leading to representational collapse.
The authors of Bootleg argue that focusing only on the input (pixels) or only on the output (final embedding) is wasteful. Instead, we should use the entire visual pathway.
Methodology: The Multi-Layer Insight
The core of Bootleg is Hidden-Self-Distillation. Instead of projecting to a single target, the predictor must reconstruct a concatenated vector derived from several blocks of the teacher (an Exponential Moving Average of the student).
Figure 1: The Bootleg framework. The student encoder sees visible patches and uses a predictor to estimate hidden representations across various depths of the teacher-encoder.
Why does this work?
- Abstraction Spectrum: Early layers provide "grounding" (edges, textures), while deep layers provide "semantics" (objects). By learning both, the model becomes more robust.
- Target Diversity: D深い (Deep) embeddings can be too abstract and correlated. Denser targets across the hierarchy act as a stronger information bottleneck, forcing more efficient compression.
- Stability: The early-layer targets are more "stimulus-driven" and less likely to drift, providing an anchor for the training process.
Rethinking Masking: The Bug Fixes
The paper reveals that I-JEPA’s success was partially hampered by implementation details. The researchers identified several "off-by-one" errors in original masking strategies where certain image edges were never seen by the encoder. Bootleg introduces a symmetric, randomized truncation strategy that ensures every part of the image has an equal chance of being "context" or "target."
Performance: Crushing the Baselines
The results are staggering for a non-contrastive, single-view method. Bootleg doesn't just edge out competitors; it leaps over them.
Table 1: Comparison across ImageNet-1K, iNaturalist, and Semantic Segmentation tasks. Bootleg (ours) consistently dominates the non-contrastive category.
Key highlights include:
- ImageNet-1K: ViT-S attentive probe hits 75.3%, outperforming standard I-JEPA by over 13 points.
- Semantic Segmentation: On ADE20K, Bootleg’s frozen backbone achieves 33.9 mIoU, significantly higher than the 21.1 of I-JEPA, proving that the multi-layer objective preserves spatial detail necessary for dense prediction.
- Low-Shot Efficiency: With only 1% of labels, Bootleg reaches 61.1% accuracy, whereas previous SOTA struggled below 50%.
Critical Analysis & Conclusion
Bootleg demonstrates that the architecture of the objective function is just as important as the model itself. By "using the whole buffalo"—extracting every bit of signal from the hidden layers—we reach a point where single-view methods can rival the performance of expensive multi-view contrastive methods (like DINO) without the massive batch size requirements.
Limitations: The teacher-encoder adds a ~12% computational overhead. However, since the distillation targets are collected in a single pass, the marginal cost of adding more targets is negligible (~2%).
Future Outlook: Bootleg shifts the focus toward hierarchical reconstruction. It opens the door for universal foundation models that can be grounded in low-level physics while reasoning with high-level semantics across any modality, from medical imaging to remote sensing.