How should task-state horizon for embodied agents be tested before it interacts with people?

Why simulation with dialogue is the first test you should run

The most practical first step is to test the agent in a simulated environment where it must converse with a human-like partner to complete tasks. The TEACh dataset, built from over 3,000 human-human dialogues in a simulated household, shows that real tasks like 'Make Coffee' or 'Prepare Breakfast' require the agent to ask questions and get extra information—not just follow a fixed plan [4]. So your test should include a human or a simulated human giving instructions that are incomplete or ambiguous, and you should measure whether the agent asks for clarification and recovers from mistakes.

Why simulation first? It lets you run many scenarios cheaply and safely, and it forces you to define the interaction protocol before you risk a physical robot. The TEACh benchmarks specifically evaluate dialogue understanding, language grounding, and task execution—three skills that are exactly what breaks in real human interaction [4]. If the agent can't handle a vague instruction in simulation, it certainly won't in the real world.

Test with a virtual avatar before a physical robot—people behave differently

A 2023 study on human-agent negotiation found that people behave differently depending on whether the agent is physically embodied or a virtual avatar on a screen. In that study, social welfare—the overall benefit to both sides—was significantly higher when participants negotiated with a virtual avatar than with a physical robot, and participants made more collaborative moves in the virtual setting [3]. This means a physical robot can actually make people more defensive or competitive, so testing with a virtual avatar first gives you a more optimistic read on the agent's social skills, and it's safer and cheaper.

But the same study found that participants perceived the physical robot as more humanlike [3]. So if your goal is to deploy a physical robot, you must eventually test with the real embodiment, because the physical presence changes the interaction. The practical takeaway: use virtual avatars for early iteration, then move to physical robots for final validation, and expect that people will take the physical robot more seriously—which can be good or bad depending on your task.

Don't forget ambiguous instructions and multi-agent coordination

Real human instructions are often ambiguous—they leave out implicit details that a person would assume. A 2022 study on embodied multi-agent task planning explicitly addressed this by having robots reason about high-level instructions like 'set the table' where some target objects are not named [5]. The study showed that agents need to use external knowledge and visual perception to resolve the ambiguity and then dynamically allocate subtasks among multiple agents [5]. So your test should include instructions that are missing information, and you should check whether the agent can fill in the gaps and coordinate with other agents if the task involves more than one.

This is especially important because the study demonstrated that the same model could work in both simulation and physical scenarios, avoiding the notorious sim-to-real gap [5]. That means you can test ambiguity handling in simulation with confidence that the results will transfer to the physical world—if you design the test to include the same kind of ambiguous instructions you expect in real use.

Measure trust carefully—explanations can cause overreliance

When you test with people, you need to measure not just task success but also trust. A 2023 study with 215 participants found that adding explanations to an AI recommendation system increased trust, but only under certain conditions [1]. Specifically, feature-importance explanations boosted trust more than counterfactual ones, and when the system's performance was not guaranteed, explanations led to overreliance—people trusted the system too much [1]. This is a warning: if your agent explains its actions, you must test whether those explanations cause people to overtrust it, especially when the agent might be wrong.

The study also found that system performance had a stronger impact on trust than explanations or risk level [1]. So the most important thing you can do is make the agent perform reliably; explanations are secondary. In your testing, include scenarios where the agent fails, and see whether people still trust it too much—that's a red flag for real-world deployment.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2022 to 2023, 1 in Q1 journals, collectively cited 1,265 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 29 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions