[Google DeepMind] A Mixed Diet Makes DINO An Omnivorous Vision Encoder

TL;DR

Researchers from Google DeepMind and UCL have transformed DINOv2 into an "Omnivorous" encoder capable of understanding RGB, Depth, and Segmentation in a single, unified feature space. By fine-tuning only the final layers of a frozen foundation model using a clever mix of contrastive alignment and semantic anchoring, the model achieves massive gains in cross-modal retrieval and even improves standard ImageNet classification, proving that structure-rich signals (like depth) help models see the world more clearly.

Background Positioning: The Gap in "Foundation" Models

While models like DINOv2 are considered "visual foundation models," they are surprisingly fragile when faced with anything other than standard RGB pixels. The authors reveal a startling reality: if you feed a depth map of a room into a standard DINOv2, the resulting features have almost zero similarity to the features of the same room's RGB photo. In the quest for true AGI-like perception, a model should exhibit perceptual stability—the internal representation of a "chair" should remain consistent whether it's perceived through color, depth, or semantic boundaries.

Problem & Motivation: The Shortcut Trap

Previous attempts at multi-modal alignment often fell into two traps:

1. Representational Collapse: If you force alignment too hard, the model forgets the rich semantic details it learned during pre-training, collapsing into a "trivial" space.
2. Low-Level Shortcuts: Models are lazy. If a depth map is always grayscale and RGB is color, the model will simply align "grayscale-ness" rather than "geometric structure."

Methodology: The Omnivorous Recipe

The authors propose a "Mixed Diet" training strategy that is both parameter-efficient and theoretically grounded.

1. The Architecture: Teacher-Student Anchoring

Instead of retraining the whole model, they freeze the first 8 blocks of a ViT-B/14 and treat the last 4 blocks as a trainable "adapter."

• Student: Learns to align modalities.
• Teacher (Frozen DINOv2): Acts as an anchor.
• The Loss: ${\mathcal{L}}_{total}={\mathcal{L}}_{align}+{\lambda }_{anchor}{\mathcal{L}}_{anchor}$. The anchoring loss ensures the student doesn't "drift" away from the powerful semantic space DINOv2 already built.

2. The Data Secret: Natural Colorization & Mixup

To stop the model from taking shortcuts, they invented Natural Colorization. They take the color palette of the RGB image and apply it to the Depth map. Now, the Depth map looks like a "textured" version of itself. Furthermore, Modality Mixup stochastically blends RGB and Depth during training. This creates a continuous spectrum (e.g., 30% RGB + 70% Depth), forcing the model to become truly modality-invariant.

Experiments & Results: Beyond RGB

The results show that "Omnivorous" training isn't just about handling depth—it actually makes the model better at visual tasks in general.

SOTA Cross-Modal Alignment

In retrieval tasks where the model must find a depth map that matches an RGB query, the Omnivorous encoder crushed the baseline.

• ScanNet R@1: Jumped from 4.6% (DINOv2) to 46.1%.
• Median Rank: Dropped from 401.8 to 2.0.

Zero-Shot Magic

The most impressive feat is training a "depth prediction head" on RGB images and then running it on Segmentation maps.

• DINOv2: Fails completely (RMSE 1.536).
• Omnivorous: Succeeds (RMSE 0.532), because it "sees" the segmentation map in the same semantic space as RGB.

Critical Analysis & Conclusion

Takeaway

The core insight is that structural alignment acts as a regularizer. By forcing the model to realize that a "depth edge" is the same as an "RGB edge," we build a more robust visual representational space. This is why the linear probe on ImageNet improved—the model's features became more "object-centric" and less "texture-dependent."

Limitations & Future Work

The current approach relies on paired data (RGB and Depth for the same scene). While these are available in simulation (MOVi) or scanned datasets (ScanNet), the next frontier is achieving this "Omnivorous" property using unpaired, "in-the-wild" data.

Closing Thought

This work signals a move away from "RGB-only" foundation models toward true sensory-agnostic encoders. In the future, we won't need different models for different cameras—we will have one omnivorous brain that understands the scene regardless of its input format.