[ICLR 2025] Reflective Test-Time Planning: Turning Robot Errors into Experience

TL;DR

Standard embodied LLMs are "brittle oracles"—they make a mistake, fail the task, and then make the exact same mistake if you reset the scene. Reflective Test-Time Planning changes this by allowing the robot to "think before it acts" (reflection-in-action) and "learn from what happened" (reflection-on-action) by updating its actual weights during deployment. This approach triples the success rate in complex household tasks and allows robots to solve spatial puzzles that stumped previous SOTA models.

Problem: The Loop of Repeated Failure

In the world of Embodied AI, we often treat Large Language Models (LLMs) as fixed policies. While they are great at high-level reasoning, they lack the "reflective practitioner" quality humans possess.

1. Textual Memory is Brittle: Methods like Reflexion store critiques in a text buffer, but this context often gets ignored or lost in long-horizon tasks.
2. Static World Models: Predicting the future is hard. If a robot's internal world model is slightly off—say, it doesn't realize a small box will block a larger one—it will keep making that bad plan forever because its parameters never change.

The authors argue that for a robot to be truly "smart," it needs to learn from the process of interaction, not just the outcome.

Methodology: The Reflection-Action Loop

The core of this paper is the execution of a "Double-Loop" learning cycle during test-time. It breaks down into three distinct phases:

1. Reflection-in-Action (Thinking Before Acting)

Instead of just outputting one action, the agent samples $N$ candidates. An Internal Reflection LLM ($V_{\varphi i}$) scores these candidates. The robot then picks the one it thinks will work best. This is essentially "test-time scaling" via deliberation.

2. Reflection-on-Action (Learning from Reality)

After the robot acts, an External Reflection LLM ($V_{\varphi e}$) looks at the result (e.g., "The box didn't fit because the shelf was too low") and provides a score.

3. Retrospective Reflection & Test-Time Training

This is the "secret sauce." Since immediate feedback is often short-sighted (an action might look good now but cause a failure 10 steps later), the agent periodically performs hindsight re-evaluation.

It then uses these hindsight scores to:

• Update the Action Policy (${\pi }_{h}eta$): Using a policy gradient (REINFORCE) to make successful actions more likely.
• Update the Internal Evaluator ($V_{\varphi i}$): Using supervised learning so the robot's future internal simulations align better with reality.

Figure 1: The dual flow of internal simulation and external training.

Experiments: Solving the Unsolvable

The authors tested this on a Long-Horizon Household benchmark and a Cupboard Fitting task.

Key Result: The "Fitting" Breakthrough

The most impressive jump was in "Fitting" tasks (e.g., packing a cupboard). These require precise spatial reasoning.

• Vanilla Baselines: Often 0-10% success.
• Reflective Test-Time Planning: 44.7% success.

Table 1: Comparison across task types. Note the massive gap in Fitting and Selection tasks.

Generalization to Reality

The model was trained on synthetic data (BEHAVIOR-1K) but tested on photorealistic HM3D scenes. It maintained a significant lead, proving that the mechanism of reflection is more robust than the visual features alone.

Critical Insight: Why Parameter Updates Matter

The authors performed a "compute-matched" experiment. They gave the baseline models 3x more time (steps) to see if they could eventually succeed just by trial and error. They couldn't. Without updating weights, the baselines simply repeated the same strategic errors. This proves that "Reflective Computation" is qualitatively different—and more valuable—than just adding more rollouts.

Figure 5: A real-robot example showing the robot correcting its placement based on retrospective feedback.

Conclusion & Future Outlook

This work marks a shift from LLMs as "static planners" to "dynamic learners." By allowing embodied agents to update their internal logic based on the "light shed by error," we move closer to robots that can be dropped into a new house and figure things out on the fly.

Limitations: The current system relies on verbal reflections, which can be biased, and the 3x latency overhead might be a challenge for high-speed dynamic tasks. However, for household logic and complex manipulation, this is a major step forward.