How should persistent 3D semantic memory for navigation be tested before it interacts with people?

What has changed: from static maps to persistent, dynamic memory

Older navigation systems assumed a static world where obstacles don't move and the map is built once. Recent work overturns this: real indoor spaces change as people interact with objects, and a persistent memory must capture those dynamics. The interactive semantic model (ISM) explicitly models how objects like chairs or doors reshape navigable space, using two levels of 'functional space' to decide what is traversable at any moment [2]. This means testing must include dynamic scenes, not just static ones.

Similarly, the transformer-based memory for interactive navigation treats the environment as a partially observable process where the agent can push obstacles aside [4]. This shows that memory isn't just about storing locations—it must also store the history of interactions to avoid getting stuck. So, any test should include cluttered, interactive environments, not just empty corridors.

What to test: benchmarks that force memory to work hard

The key is to use benchmarks that require the agent to remember previously seen objects across multiple goals. The multiON task does exactly this: it asks the agent to navigate to a sequence of objects in a realistic environment, and results show that performance degrades dramatically as the sequence length increases [1]. This is a direct test of memory persistence—if the agent forgets where it saw the first object, it fails later goals.

For cross-floor navigation, the HM3D-MFMON benchmark adds a floor transition requirement, forcing the agent to use its 3D memory to navigate stairs and find objects on different floors [3]. The study found that a persistent 3D semantic voxel memory outperforms a planar map baseline, proving that 3D memory is essential for multi-floor tasks. So, your test should include multi-floor scenarios to validate the 3D aspect of memory.

The catch: generalization to unseen environments is still hard

Even the best memory systems struggle in unseen environments. The Bayesian Relational Memory (BRM) approach explicitly targets this by learning a probabilistic graph of room layouts, and it outperforms baselines that don't use such structure [5]. This suggests that testing should include unseen environments to check if the memory generalizes beyond the training data.

However, there's a sobering finding: even oracle map agents—which have perfect maps—achieve only relatively low performance on multiON [1]. This means that memory alone isn't enough; the navigation policy must also be effective. So, when testing, don't just measure success rate; also analyze where failures occur (e.g., planning vs. perception) to identify bottlenecks.

About These Sources

This answer is built on 5 studies (2 peer-reviewed, 3 preprints) — published from 2022 to 2026, 1 from 2024 or later, 1 in Q1 journals, collectively cited 74 times — selected as the most relevant from 6 studies that passed quality screening, drawn from 50 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions