InterLV-Search: When Visual Cues Become the Compass of Agentic Search

TL;DR

InterLV-Search is a new frontier in multimodal AI evaluation. While previous benchmarks asked models to "look at this picture and answer a question," InterLV-Search asks models to "search the web, find a picture, use a clue in that picture to decide which website to visit next, and repeat." It reveals that even the world's best MLLMs fail to maintain coherence when visual evidence serves as a pivot for search logic.

The Problem: The "Visual Endpoint" Fallacy

In current multimodal research, we often treat images as the destination. We search for a specific entity and then perform Visual Question Answering (VQA) to extract a color, a count, or a label.

However, human researchers don't work like that. If you are researching a historical artifact, you find one photo, notice a specific maker's mark (a logo), and that visual mark determines your next search query. The image is not the end; it is a pivot.

Existing benchmarks like VisBrowse or MMSearch largely ignore this "search-controlling" role. InterLV-Search fills this void by requiring Recurrent Vision–Text Interleaving.

Methodology: A Three-Tiered Ladder of Complexity

The benchmark is structured into three levels of increasing difficulty:

1. Level 1: Active Visual Evidence Seeking: Can the agent infer it needs an image and go find it?
2. Level 2: Controlled Offline Interleaved Search: Can the agent handle multi-hop transitions where an image provides the bridge to the next entity in a closed Knowledge Graph?
3. Level 3: Open-Web Interleaved Search: The "Wild West" of search. Agents face noisy webpages and must handle Multi-branch logic—comparing evidence from multiple visual sources before deciding which path to follow.

Figure: The data construction pipeline utilizing Knowledge Graphs (MMKG-W) and MLLM-driven generation.

InterLV-Agent: The Standardized Playground

To make evaluation fair, the authors introduced InterLV-Agent. This framework provides:

• Unified Tooling: Web search, Image search, Reverse image search, and Image cropping.
• Dual-Layer Memory: Short-term memory for recent logs and long-term memory for compact history notes.

Experimental Insights: We Are Far from Solving This

The results are a wake-up call for the LLM community.

• Proprietary Dominance: Models like Gemini-3.1-Pro and GPT-5.4 lead the pack, yet their overall accuracy on Level 3 Multi-branch tasks is only ~33-37%.
• Open-Source Struggle: Specialty search agents (like SenseNova-MARS) showed almost no gain—and sometimes even performance degradation—when given access to tools on this benchmark, indicating poor planning capabilities.
• The Localization Bottleneck: Models often fail not because they can't reason, but because they can't find the specific visual evidence required (visual retrieval recall is low).

Table: Performance of various MLLMs. Note the sharp drop in "Multi-Branch" vs. "Single-Chain" accuracy.

A Case Study in Failure

The paper highlights a "Multi-Branch" task involving a Tate gallery page and a Berlinale festival page. The model failed because it relied on broad textual strings rather than "grounding" the visual symbols (emblems) mentioned in the prompt. It searched for words but didn't "look" at the pivots needed to count the specific icons required for the next step.

Figure: A successful interleaved trajectory through cities (Hangzhou → Wuhan → Galați → Mumbai).

Conclusion and Future Outlook

InterLV-Search proves that "interleaving" is the hard part of multimodal intelligence. Future "Deep Research" agents will need much better visual-spatial grounding and long-horizon planning to follow the breadcrumbs hidden within images across the web. This benchmark provides the necessary map for that journey.