What RoboGraph's episodes measure—and what they leave out
RoboGraph's task-state horizon (TSH) episodes are designed to test how well an agent tracks the evolving state of the world—like remembering that a door was opened or that an object moved—across long tasks. The benchmark includes 588 episodes across 84 scenes and found that most of the 15 tested agentic models struggle with demanding TSHs [1]. That tells you a lot about state tracking, but it doesn't tell you whether the agent knows when it's actually done. A separate evaluation framework, VIGIL, showed that standard benchmarks collapse three distinct failures into one: never completing the task, completing it but failing to stop, and reporting success without enough evidence [5]. In VIGIL's tests, models with comparable world-state completion scores differed by up to 19.7 percentage points in benchmark success—meaning one model correctly reported completion while another with nearly identical execution drifted past the goal without closing [5]. So RoboGraph's episodes can show you that an agent tracked states, but they won't reveal whether the agent can correctly commit to task completion at the end.
What this means for benchmarking embodied agents
If you're using RoboGraph to evaluate an agent, you should treat its TSH scores as a measure of state-tracking ability, not overall task competence. The evidence here suggests that state tracking is necessary but not sufficient: an agent can track states perfectly and still fail to terminate correctly, use memory poorly, or lose alignment between its internal plan and reality. The PhyBlock benchmark, which tests VLM-based agents on 3D block assembly, found that performance degrades substantially as task complexity increases, reflecting weaknesses in handling spatial dependencies and multi-step physical constraints [3]. That aligns with RoboGraph's finding that most models struggle with demanding TSHs, but it also points to a broader issue: current agents lack grounded representations of physical structure and consistent reasoning across sequential steps [3]. So RoboGraph's episodes are a valuable diagnostic for state tracking, but to get a full picture of an agent's long-horizon capabilities, you'd need to combine them with tests for terminal commitment, memory use, and alignment—like those from VIGIL, WorldLines, and ContextFlow.
About These Sources
This answer is built on 5 studies (all preprints) — published in 2026, 5 from 2024 or later — selected as the most relevant from 5 studies that passed quality screening, drawn from 30 papers retrieved from a database of over 500 million.
Sources used in this answer
Compiling and Benchmarking Task-State Horizons for Embodied Agents
RoboGraph, a benchmark of 588 episodes across 84 scenes, found that most of 15 advanced agentic models struggle with demanding task-state horizons, revealing gaps in maintaining, exploring, and updating task-relevant state over long horizons.
ContextFlow: Hierarchical Task-State Alignment for Long-Horizon Embodied Agents
ContextFlow identifies task-state misalignment—where the planner's active stage, runtime evidence, remembered context, and delegated executor no longer justify the same next-step decision—and proposes scoped updates to mitigate failures like unsupported handoffs and stage lock.
Learning and Reasoning for Embodied Agents in Long-Horizon Decision-Making Tasks
PhyBlock, a benchmark of 2,600 tasks (400 assembly, 2,200 VQA), found that 23 state-of-the-art VLMs show significant limitations in maintaining coherent reasoning and effective planning over extended horizons, with performance degrading substantially as task complexity increases.
WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents
WorldLines, a project-driven benchmark for long-horizon household assistance, found persistent challenges in partial observability, overwritten world states, and translating long-term memory into embodied plans, even with a proposed observer-grounded memory framework.
Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents
VIGIL, an evaluation framework that separates world-state completion from terminal commitment, found that models with comparable world-state completion scores differed by up to 19.7 percentage points in benchmark success, revealing that some models drift past the goal without correctly closing the episode.