[CVPR 2026] C2FMAE: Bridging the Gap Between Semantics and Pixels via Hierarchical Masked Autoencoders
From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding
C2FMAE is a hierarchical self-supervised pre-training framework that integrates semantic masks, instance masks, and RGB pixels. By employing a cascaded decoder and a progressive masking curriculum, it achieves SOTA results across ImageNet classification (84.2% Top-1) and downstream dense prediction tasks.
TL;DR
C2FMAE (Coarse-to-Fine Masked Autoencoder) resolves the "attention drift" in self-supervised learning by forcing the model to learn in a top-down hierarchy. By reconstructing Semantic Masks → Instance Masks → RGB Pixels in a cascaded fashion, it achieves superior performance in both high-level classification and low-level dense prediction tasks with 4x higher training efficiency than standard MAE.
Context: The Paradox of Visual Representation
In the current SSL landscape, we have two dominant but bifurcated paths:
- Contrastive Learning (CL): Focuses on global semantics (the "what") but ignores the "where" and fine-grained textures.
- Masked Image Modeling (MIM): Focuses on local reconstruction (the "how") but often gets lost in background noise due to semantically-agnostic random masking.
The authors identify this as "Attention Drift". While humans see a "forest before the trees," standard MIM treats every pixel with equal importance, leading to diffuse attention maps that fail to recognize object boundaries effectively.
Methodology: The Coarse-to-Fine Principle
C2FMAE introduces three layers of granularity into a unified Vision Transformer (ViT) framework.
1. Cascaded Decoder: Sequential Refinement
Unlike MultiMAE, which uses a parallel decoder (treating modalities as independent), C2FMAE uses a Cascaded Decoder. The output of the Semantic stage feeds into the Instance stage, which then feeds into the RGB stage. This forces the model to use high-level "concepts" to guide the generation of "objects" and eventually "pixels."
2. Progressive Masking Strategy
To implement a "learning curriculum," the authors dynamically shift the masking focus during the pre-training epochs:
- Semantic-guided Phase: Masks are distributed based on semantic regions (focusing on scene layout).
- Instance-guided Phase: Masks prioritize object regions (focusing on thing-vs-stuff).
- Random Masking Phase: Standard uniform masking to capture fine-grained textures.
Experiments and Results
The model was pre-trained on a newly constructed multi-granular ImageNet-1K dataset (using Grounded-SAM pseudo-labels).
Performance Highlights:
- ImageNet-1K: 84.2% Top-1 Accuracy (ViT-B).
- Efficiency: A 400-epoch C2FMAE model outperforms a 1600-epoch MAE model, illustrating that hierarchical guidance provides a much stronger inductive bias than raw pixel reconstruction.
- Dense Prediction: On COCO and ADE20K, C2FMAE shows significant gains (+1.8 APb / +1.3% mIoU), proving that its learned features are more spatially aware.
Deep Insight: Beyond Pixels
The visualization of the model's generative capability (predicting pixels from only instance or semantic masks) reveals a fascinating property: the model essentially learns to "disentangle" geometry from texture. As shown in the qualitative results, C2FMAE can reconstruct sharp object boundaries even when given sparse, abstract inputs—a feat standard MAE struggles with.
Critical Analysis & Conclusion
Takeaway: C2FMAE proves that "more data modalities" isn't enough; you need a "hierarchical structure" to fuse them. The transition from parallel decoders to cascaded decoders is a subtle but profound shift in how we think about multi-modal SSL.
Limitations: The framework currently relies on high-quality pseudo-labels from teacher models (SAM, SEEM). Future iterations might explore how to evolve these hierarchies in a completely "bottom-up" self-supervised manner without pre-existing segmentation models.