Why task-state horizon is the real bottleneck, not action length
Most robotic benchmarks measure difficulty by how many actions a task takes or how many subtasks it has. But that misses a distinct challenge: agents must track task-relevant world states that change as they explore and as the environment changes on its own. The concept of task-state horizon (TSH) captures this—the span of state transitions an agent must remember and update. In a benchmark of 588 episodes across 84 scenes, 15 advanced agentic models showed substantial performance drops as TSH increased, even when action sequences were similar [1]. This means a long action list isn't the hard part; keeping track of what changed and what still matters is.
The practical implication: if your workflow only optimizes for completing steps, you'll build agents that fail in real homes where objects move, devices toggle, and unexpected events occur. The RoboGraph compiler specifically constructs TSH from spatial and temporal causal dependencies, including failures and interventions during execution [1]. That's a signal to stop treating state tracking as an afterthought and start measuring it explicitly.
Redesign your benchmarks to inject state changes and partial observability
Current benchmarks often assume the agent sees everything it needs at each step. But real environments are partially observable—you can't see behind a door or know if a device is on without checking. The WorldLines benchmark builds temporally extended household traces with dialogues, actions, execution feedback, and object/device state changes, then tests agents on memory QA and embodied task planning [2]. It found persistent challenges in partial observability and overwritten world states—meaning agents forget or misremember what changed. If your workflow doesn't include such dynamic state changes, you're not testing the real problem.
Concretely, add episodes where the environment changes without the agent's action (e.g., a door closes, a light turns off) and where the agent must infer state from incomplete observations. The RoboGraph benchmark already does this by including unexpected failures and interventions [1]. Use these to measure not just final success but whether the agent's internal state representation stays accurate over time.
Adopt memory architectures that explicitly track state, not just past observations
A common failure is that agents treat memory as a list of past observations, but they don't update a coherent model of the current world state. The Scene Memory Transformer (SMT) embeds each observation into a memory and uses attention to exploit spatio-temporal dependencies, outperforming reactive and memory-based policies on visual navigation tasks [3]. That shows the value of a structured memory that can be queried for relevant past information.
The WorldLines paper goes further by proposing ObsMem, an observer-grounded memory framework that maintains visibility-aware memories and action-native state trails [2]. This means the agent remembers what it could actually see and tracks the state changes caused by its own actions. In tests, this framework offered a stronger reference architecture for state-aware decisions, though challenges remained. For your workflow: don't just feed raw history into a transformer; design memory modules that update a state estimate and handle uncertainty about what's visible.
About These Sources
This answer is built on 3 studies (all preprints) — published from 2022 to 2026, 2 from 2024 or later — selected as the most relevant from 3 studies that passed quality screening, drawn from 44 papers retrieved from a database of over 500 million.
Sources used in this answer
Compiling and Benchmarking Task-State Horizons for Embodied Agents
Introduced task-state horizon (TSH) and RoboGraph, a compiler that builds benchmarks with spatial/temporal causal dependencies including failures and interventions; across 588 episodes and 84 scenes, 15 advanced agentic models struggled with demanding TSHs, showing gaps in maintaining, exploring, and updating state.
WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents
WorldLines benchmark constructs long-horizon household traces with dialogues, actions, feedback, and state changes; found persistent challenges in partial observability and overwritten world states, and proposed ObsMem, an observer-grounded memory framework that improved state-aware decisions.
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Scene Memory Transformer (SMT) embeds observations into a memory and uses attention to exploit spatio-temporal dependencies, outperforming reactive and memory-based policies on visual navigation tasks.