What does 'task-state horizon' really mean for an embodied agent?
Task-state horizon refers to the length and complexity of the sequence of states an agent must track and act upon to complete a task. A fair evaluation must measure not just the number of steps, but the agent's ability to maintain a coherent understanding of the task state over time, including what has been done, what remains, and how to adapt when things change. For example, TEACh tasks range from 'Make Coffee' to 'Prepare Breakfast', with the latter requiring many more steps and more dialogue to resolve ambiguity [4]. This shows that horizon is not just about duration but about the accumulation of decisions and interactions.
FurnitureBench explicitly targets 'long-horizon complex manipulation' and provides 5,000+ demonstrations of furniture assembly, which requires planning, dexterous control, and visual perception over many steps [5]. The benchmark's design forces agents to handle a sequence of sub-tasks, each with its own state, and to integrate them into a coherent whole. Thus, a fair evaluation must include tasks that require sustained attention and memory, not just single-step actions.
Why memory and reasoning are the core of horizon evaluation
A fair evaluation must test the agent's memory of visited states and its ability to reason over them. In Knowledge-based Embodied Question Answering (K-EQA), agents must explore an environment and answer questions like 'what are objects used to cut food in the room?' which requires recalling what they saw and applying external knowledge [2]. The 3D scene graph in that work provides a memory store, and the authors show that this significantly improves efficiency for multi-turn question answering [2]. This suggests that horizon evaluation should measure how well an agent retains and uses information from earlier in the task.
Similarly, ERRA uses a coarse-to-fine inference approach to decompose long-horizon language-conditioned manipulation tasks, with the coarse level generating action language and the fine level executing actions [3]. The feedback from action execution feeds back into the coarse reasoning, showing that horizon evaluation must capture the agent's ability to update its plan based on new observations. Without measuring this iterative reasoning, a benchmark would miss the essence of long-horizon tasks.
What benchmarks exist and what do they tell us about current agents?
The most comprehensive benchmark here is TEACh, with over 3,000 human-human interactive dialogues for household tasks [4]. It provides three benchmarks for dialogue understanding, language grounding, and task execution, and initial models still struggle, indicating that current agents are far from mastering long-horizon interactive tasks [4]. This is a key point: a fair evaluation must include interactive dialogue, because real-world tasks often require asking for clarification or correcting mistakes.
FurnitureBench is the largest real-world benchmark for long-horizon manipulation, with 200+ hours of pre-collected data and 5,000+ demonstrations [5]. The authors benchmark offline RL and imitation learning algorithms and find that these need improvement to solve the tasks in the real world [5]. This shows that even with abundant data, current methods fall short, so a fair evaluation must be challenging enough to expose these limitations. Both benchmarks agree that current agents are not yet capable of robust long-horizon performance, which is a crucial baseline for any evaluation.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2022 to 2024, 1 from 2024 or later, 2 in Q1 journals, collectively cited 172 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 32 papers retrieved from a database of over 500 million.
Sources used in this answer
Embodied Intelligence Toward Future Smart Manufacturing in the Era of AI Foundation Model
This position paper defines embodied intelligence and discusses challenges for implementation, but does not provide empirical data on task-state horizon evaluation.
Knowledge-Based Embodied Question Answering
In the K-EQA task, the proposed framework uses a 3D scene graph as memory to improve multi-turn question answering efficiency, demonstrating that memory is critical for long-horizon embodied reasoning.
ERRA: An Embodied Representation and Reasoning Architecture for Long-Horizon Language-Conditioned Manipulation Tasks
ERRA combines coarse and fine inference to handle long-horizon language-conditioned manipulation, showing that iterative feedback between planning and execution is necessary for such tasks.
TEACh: Task-Driven Embodied Agents That Chat
TEACh provides over 3,000 human-human dialogues for household tasks, with benchmarks for dialogue understanding, language grounding, and task execution, and initial models perform poorly, indicating the difficulty of long-horizon interactive tasks.
FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation
FurnitureBench offers 5,000+ demonstrations of furniture assembly and benchmarks offline RL and IL algorithms, finding that these methods need improvement to solve real-world long-horizon tasks.