Do visual retrievers really outperform text-based ones when documents change?
Yes, and the gap is substantial. The MMDocIR benchmark, which includes 1,685 expert-annotated questions and over 173,000 bootstrapped labels, found that visual retrievers significantly outperform their text counterparts in both page-level and layout-level retrieval tasks [1]. This means that when a document's text changes—say, a table gets updated or a figure is replaced—a visual retriever that captures the entire page layout is less likely to lose track of relevant content. The benchmark also showed that text retrievers using vision-language model (VLM) generated text did much better than those relying on OCR text, suggesting that OCR is a weak link when documents evolve [1].
The Document Screenshot Embedding (DSE) approach takes this further by removing the need for any content extraction preprocessing. DSE treats document screenshots as a unified input, preserving all text, images, and layout information. In tests on a 1.3 million Wikipedia page corpus, DSE outperformed BM25 by 17 points in top-1 retrieval accuracy—meaning it found the right page on the first try far more often [2]. In a mixed-modality slide retrieval task, DSE beat OCR-based text retrieval by over 15 points in nDCG@10, a standard ranking metric [2]. These results are consistent across different document types, showing that visual methods are more robust to collection changes because they don't depend on fragile text extraction pipelines.
Does breaking documents into chunks hurt reliability when collections grow?
Yes, segmenting documents into separate passages for retrieval causes information loss, especially as collections change. The Unified Multimodal Interleaved Document Representation approach directly addresses this by embedding entire documents—including text, images, and tables—into a single representation using vision-language models [5]. This holistic method outperformed relevant baselines in diverse retrieval scenarios, including both text and multimodal queries [5]. The key insight is that when a document is updated (e.g., a new paragraph is added or an image is swapped), a passage-level retriever may miss the new content or retrieve outdated chunks, whereas a unified representation captures the full document context.
This aligns with findings from the RAG Beyond Text study, which showed that preserving positional information of images alongside text—rather than treating them as separate chunks—improves retrieval accuracy for both OCR-compatible and OCR-incompatible images [3]. Their system achieved state-of-the-art performance on complex queries across research papers, manuals, and guides, and notably outperformed GPT-4 Vision on retrieving images that are visually or semantically misaligned with surrounding text [3]. This suggests that unified representations are not just more reliable for text changes, but also for visual content changes.
How much can we trust these results across different document types?
The evidence is strong but not universal. The MMDocIR benchmark is the most comprehensive here, covering page-level and layout-level retrieval with a large dataset, and its conclusions are consistent across tasks [1]. The DSE study tested on both text-intensive Wikipedia pages and mixed-modality slide decks, showing broad applicability [2]. The Unified Multimodal approach was tested on diverse IR scenarios including multimodal queries [5]. However, the RAG Beyond Text study focused specifically on image retrieval within RAG systems, and while it achieved state-of-the-art results, its scope is narrower [3]. The survey on Multimodal RAG [4] confirms that cross-modal alignment and reasoning remain open challenges, meaning reliability is not guaranteed for every new document type or query format. In short, the best current evidence points to visual-first, unified approaches as the most reliable, but ongoing evaluation on new benchmarks is essential.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2024 to 2025, 5 from 2024 or later — selected as the most relevant from 5 studies that passed quality screening, drawn from 43 papers retrieved from a database of over 500 million.
Sources used in this answer
MMDocIR: Benchmarking Multimodal Retrieval for Long Documents
The MMDocIR benchmark (1,685 expert-annotated questions, 173,843 bootstrapped labels) shows visual retrievers significantly outperform text retrievers in page-level and layout-level retrieval, and text retrievers using VLM-generated text outperform those using OCR text.
Unifying Multimodal Retrieval via Document Screenshot Embedding
Document Screenshot Embedding (DSE) treats document screenshots as a unified input, outperforming BM25 by 17 points in top-1 retrieval accuracy on a 1.3M Wikipedia corpus and beating OCR text retrieval by over 15 points nDCG@10 in slide retrieval.
RAG Beyond Text: Enhancing Image Retrieval in RAG Systems
A novel RAG image retrieval method that preserves positional information of images alongside text achieves state-of-the-art performance on complex queries and outperforms GPT-4 Vision on OCR-incompatible images.
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
A comprehensive survey on Multimodal RAG identifies cross-modal alignment and reasoning as key challenges, and reviews training strategies, robustness enhancements, and agent-based approaches for reliable multimodal retrieval.
Unified Multimodal Interleaved Document Representation for Retrieval
A unified multimodal interleaved document representation that embeds entire documents (text, images, tables) using vision-language models outperforms relevant baselines in diverse retrieval scenarios, including both textual and multimodal queries.