[CVPR Style] ViCLIP-OT: Architecture and Optimal Transport for Vietnamese Multimodal Retrieval
ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport
ViCLIP-OT is the first foundation vision-language model specifically optimized for Vietnamese image-text retrieval. It integrates a dual-encoder architecture (DINOv3 and Vietnamese SBERT) with a novel Similarity-Graph Regularized Optimal Transport (SIGROT) loss, achieving state-of-the-art performance on Vietnamese benchmarks.
Executive Summary
TL;DR: ViCLIP-OT is a novel foundation model that brings state-of-the-art Image-Text retrieval to the Vietnamese language. By combining the standard CLIP contrastive framework with a Similarity-Graph Regularized Optimal Transport (SIGROT) loss, the model moves beyond simple pairwise matching to capture the global structural relationships of data within a batch.
Context: Most Vision-Language Models (VLMs) suffer from a "low-resource" penalty when applied to languages like Vietnamese. This work is a significant "SOTA-setter" for the Vietnamese AI ecosystem, providing a robust backbone for intelligence multimedia systems.
The Problem: The "Modality Gap" in Low-Resource Settings
Standard CLIP-style training optimizes for Instance-level Alignment: it wants this image to be close to this caption. However, it ignores Distributional Consistency. In a training batch, multiple images might share similar semantic concepts (e.g., "sunset" or "street food"), but the standard contrastive loss treats every negative sample equally.
This leads to the Modality Gap—a phenomenon where image embeddings and text embeddings reside in completely different regions of the latent space. In low-resource scenarios like Vietnamese, where data is scarce, this gap becomes even harder to bridge.
Methodology: Bridging the Gap with SIGROT
The core innovation of ViCLIP-OT is the SIGROT loss. Instead of relying solely on the contrastive objective, it introduces an Optimal Transport (OT) layer.
1. The Architecture
The model utilizes two heavy-weight encoders:
- Vision: A DINOv3-based Vision Transformer (ViT).
- Text: A Vietnamese-specific Sentence-BERT (SBERT).
2. The Optimal Transport Mechanism
The authors define a similarity graph () that calculates relationships not just between and , but also to and to .
The SIGROT loss then solves an Unbalanced Optimal Transport (UOT) problem. This allows the model to:
- Find a "transport plan" that maps the distribution of images to the distribution of text.
- Use the "Similarity Graph" as a soft target to ensure that the matching respects the intrinsic structure of the data.
- Relax marginal constraints (using KL-divergence) to handle noisy data common in low-resource web-crawled datasets.
Experimental Performance: Deep Dive
The model was evaluated on three major Vietnamese benchmarks: UIT-OpenViIC, KTVIC, and Crossmodal-3600.
| Model | Avg R@K (UIT) | Avg R@K (XM3600) |
|---|---|---|
| CLIP Baseline | 61.59% | 45.13% |
| ViCLIP-OT | 67.34% | 56.85% |
| ViSigLIP-OT | 68.96% | 56.17% |
The improvements are particularly massive in the Zero-Shot evaluation (XM3600), where ViCLIP-OT outperformed CLIP by over 11%. This proves that the OT-based regularization significantly improves the model's ability to generalize to unseen data distributions.
Visualizing the Modality Gap
The UMAP visualizations highlight the physical effect of SIGROT. In the baseline SigLIP, image and text clusters are visibly separated. In the OT-enhanced version, the clusters interleave much more tightly, indicating a truly shared semantic space.
Critical Insight: Why Optimal Transport?
The "Secret Sauce" here is that Optimal Transport provides a Global Perspective. While CLIP looks at "Points," OT looks at "Manifolds." By forcing the image manifold to match the text manifold according to a precomputed similarity graph, the model inherits the structural knowledge of a larger multimodal embedding model (like Qwen3-VL used for the graph construction) while specializing in the nuances of the Vietnamese language.
Conclusion & Key Takeaways
- Vietnamese SOTA: ViCLIP-OT is the first robust foundation model for the region.
- OT is a powerful regularizer: It mitigates the modality gap more effectively than standard contrastive losses.
- Low-Resource Roadmap: The methodology provides a blueprint for adapting VLMs to other underrepresented languages using graph-based structural alignment.
Future Work: The authors suggest end-to-end learning of the similarity graph and expanding the framework to Visual Question Answering (VQA).