How should agentic memory for long-horizon robot manipulation be tested before it interacts with people?

Why simulation is the first testing ground—and what to measure

Simulation lets you stress-test memory cheaply and safely before any physical robot is involved. The key is to design tasks that force the agent to use memory, not just react to the current view. For instance, one study introduced a benchmark with 56 tasks across multiple environments, specifically to evaluate multistep reasoning and linguistic variation [4]. This lets you see if the agent can plan, reflect, and revise actions in a closed loop—skills that are essential for long-horizon tasks.

When you run these simulations, track success rates and efficiency. In one simulation, a memory-enhanced policy boosted success from 32.4% to 84.0% compared to a baseline without memory [1]. That's a huge jump, showing that memory is not just a nice-to-have but a critical component. Also, measure inference latency and GPU memory usage—if the memory system slows the robot down, it won't be practical in real time [1].

Real-robot benchmarks: test under perceptual aliasing and occlusion

Simulation can't capture all the messiness of the real world, so you must move to physical robots. The most important thing is to create scenarios where the same observation could come from different histories—this is called perceptual aliasing. One study built a real-robot dataset (Camo-Dataset) on a UR5e arm that includes episodic recall, spatial tracking, and sequential manipulation under such conditions [3]. This forces the robot to use memory to disambiguate what it sees.

In real-robot tests, memory systems have shown high success rates—up to 98.7% in one study [1]. But don't just look at success; also test how the robot handles distractions and memory decay. Another study found that a proactive memory agent, which decides when to inject reminders, improved pass@1 by +8.3 percentage points on a terminal benchmark and +6.8 on another, compared to passive memory [5]. This suggests that testing should include whether the robot actively recalls relevant past events at the right time, not just stores them.

Test memory decay and proactive recall—not just storage

A common failure mode in long-horizon tasks is 'behavioral state decay'—when important information gets buried or forgotten as the trajectory grows [5]. So your testing should explicitly check whether the robot can surface decision-relevant past events when needed. One study showed that a memory agent that selectively intervenes (injects a reminder) outperforms passive memory exposure or always-on injection [5]. This means you should test not just whether the robot can store memories, but whether it can retrieve them at the right moment.

Also, test how the robot handles memory compression. One approach compresses each historical frame into a single token, allowing the policy to attend over long-term history with negligible latency [1]. This is important because if memory is too heavy, the robot can't react in real time. So in your testing, measure both the accuracy of recall and the computational overhead.

About These Sources

This answer is built on 5 studies (2 peer-reviewed, 3 preprints) — published from 2024 to 2026, 5 from 2024 or later — selected as the most relevant from 7 studies that passed quality screening, drawn from 42 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions