Why aren't these benchmarks saturated yet?
Current benchmarks are still too hard for agents to master. In the MultiON benchmark, which requires navigating to a sequence of objects, performance drops dramatically as task complexity increases—even when agents are given an oracle map (perfect memory of the environment) [3]. This means the bottleneck isn't memory storage; it's how agents use memory to plan and re-locate objects. So, saturation is not imminent because the core challenge—efficiently exploiting memory—remains unsolved.
Adding language instructions makes it even harder. In the Pasture benchmark, which tests zero-shot language-driven object navigation, simple CLIP-based agents often fail to leverage language descriptions, though they can find uncommon objects [2]. This shows that even with strong memory, agents struggle to connect language to visual memory, a key skill for real-world use.
What comes next after memory benchmarks?
The next frontier is integrating memory with other navigation skills, like cross-floor movement and sequential object search. The LifelongCrossNav framework shows that combining a shared 3D semantic memory with support-aware traversability mapping and stair-specific perception enables agents to handle multi-floor, multi-object tasks that single-floor benchmarks ignore [1]. This suggests future benchmarks will need to include these real-world complexities to push progress.
Another direction is improving language grounding in memory. The CoW (CLIP on Wheels) study found that a simple agent with classical exploration and CLIP-based object localization can match a state-of-the-art trained model on zero-shot navigation, but it still struggles with spatial and appearance attributes [2]. So, next steps involve making memory more language-aware, so agents can find objects described by attributes like 'red chair' or 'behind the sofa'.
Who benefits from these advances, and when?
Robotics and embodied AI researchers benefit most immediately, as these benchmarks guide the design of navigation systems for home robots and autonomous drones. For example, the MultiON benchmark's finding that simple semantic maps outperform complex neural feature maps suggests that memory design should prioritize interpretability and efficiency [3]. This could lead to more practical robots that don't need massive compute.
End users will benefit when agents can reliably find multiple objects across floors—like a home robot fetching items from different rooms. The LifelongCrossNav benchmark, which requires at least one floor transition in its hardest subset, directly targets this use case [1]. However, given the current performance gaps, expect this to take several more years of research before it's robust enough for consumer products.
About These Sources
This answer is built on 3 studies (1 peer-reviewed, 2 preprints) — published from 2022 to 2026, 1 from 2024 or later, collectively cited 165 times — selected as the most relevant from 3 studies that passed quality screening, drawn from 38 papers retrieved from a database of over 500 million.
Sources used in this answer
LifelongCrossNav: Persistent 3D Semantic Memory for Cross-Floor Multi-Object Navigation
LifelongCrossNav introduces a persistent 3D semantic voxel memory and cross-floor traversability mapping, outperforming a planar baseline on the new HM3D-MFMON benchmark for sequential multi-floor multi-object navigation.
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
In the Pasture benchmark for language-driven zero-shot object navigation, a simple CLIP-based agent with classical exploration matches a state-of-the-art trained model's efficiency, but struggles to leverage language descriptions for spatial and appearance attributes.
MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation
The MultiON benchmark shows that navigation performance degrades dramatically with task complexity, and even oracle map agents achieve relatively low success, indicating that map-based memory alone is insufficient for long-horizon tasks.