Can test-time spatial scaffolding improve fine robotic manipulation with frozen policies?

The bottleneck is what the policy can see

The paper starts from a specific diagnosis rather than a new policy. Frontier multimodal foundation models can already generate robot actions directly from images and instructions, but they remain weak on fine manipulation: insertion, stacking, or placing an object on a narrow structure. The authors argue the limiting factor is often not policy capability but spatial observability — whether the fixed physical cameras actually reveal the relationships that decide success, such as whether a plug is aligned with its socket or a hole is centered over a peg.

That distinction matters because it changes what a fix should look like. If the problem were capability, the answer would be training or fine-tuning. If the problem is observability, the answer can sit outside the model: change what the model is shown at test time, while leaving the policy and the physical sensing setup untouched. SpatialHarness is built on that second premise, and the experiments are designed to test it by holding the policy fixed.

A simulated scene kept aligned with execution

SpatialHarness is a test-time embodied harness: software that wraps a frozen policy and structures what it observes and how it acts. Its core asset is an online simulated scene that is kept synchronized with real-world execution. Before a task, the harness builds object CAD models from multiple images and annotations, then places them by comparing rendered contours and depth against segmentation masks and observed depth from the initial real observation. A model also writes constraint programs — for example, that a Hanoi peg must lie above the horizontal support surface — which check or adjust candidate poses.

Keeping that scene aligned during contact is the hard part, and it is where the paper's interaction-aware synchronization comes in. Each object is assigned one of three modes. Static objects keep their pose, or are re-aligned visually if movable. Held objects have their pose predicted through the physics engine using measured joint and gripper motion, then validated visually and against object–gripper geometry, with a wrist-camera estimate substituted if either check fails. Transition mode covers grasp and release: instead of correcting only the post-interaction pose, the harness saves the pre-interaction state, perturbs it into hypotheses, replays the same measured robot motion from each, and selects the full state most consistent with real execution and wrist-camera observations.

Views are chosen for the task, then held fixed

The second component decides what the policy actually sees. Guided by a multistep reasoning prompt, the model derives the spatial constraints the objects must satisfy at task completion. If those constraints concern the relative positions and orientations of whole objects, it selects two global virtual viewpoints; if they concern how local structures fit together, it selects one local and one global view. The model then renders candidates and checks whether the relevant parts are simultaneously visible, whether the critical relationship can be clearly assessed, and whether the view reveals something the current real observation does not. Unsuitable viewpoints get small adjustments and re-rendering.

During execution the virtual viewpoints stay fixed while the harness keeps rendering from the continually updated simulated scene. The policy receives these complementary virtual views together with structured spatial information, including object 6-DoF poses and estimates of rendering reliability, alongside the task instruction, real observations, and robot states. The paper separates two roles for the same underlying model: it acts as the model when reasoning about constraints and configuring viewpoints, and as the policy when generating actions through tool calls.

Success rises on all four real-robot tasks

The evaluation uses a Franka Research 3 arm and four real-robot tasks spanning precise geometric alignment, object-relative placement, and articulated-object interaction: block stacking, plug insertion, Tower of Hanoi, and opening a drawer followed by block placement. Each task and method gets 15 trials, with the manipulation policy and control protocol held fixed. Average success rises from 18.33% to 83.33%, a gain of 65.00 percentage points.

The per-task pattern is worth separating, because the tasks start from different baselines. On Tower of Hanoi and the drawer task the baseline fails all 15 trials, while SpatialHarness succeeds in 15 and 10 respectively. The authors attribute this to the original observations not clearly revealing the hole–peg offset or the gripper–handle contact, so the added views supply information the key alignment and grasping steps needed. Plug insertion is the most precision-demanding task, requiring alignment in both position and orientation; there the baseline already succeeds in some trials at 26.67%, and SpatialHarness raises this to 66.67%, a 40.00-point gain. Block stacking moves from 46.67% to 100%. The last two cases show the added views helping even where the real cameras already supported occasional success, by making subtle misalignments visible.

More successes without more policy queries

A separate measure answers whether the gains came from simply asking the policy more often. They did not. The number of queries to the manipulation policy either decreased or stayed largely unchanged while success rose on all four tasks. For block stacking and Tower of Hanoi, policy queries fell by roughly 30% and 21% respectively, with both tasks reaching 100% success. Plug insertion gained 40.00 percentage points while queries fell about 7%. On the drawer task, queries were comparable to the baseline while success rose from 0% to 66.67%.

The authors interpret this as auxiliary virtual rendering reducing ineffective attempts and repeated adjustments caused by limited spatial observability, which in turn lowers query overhead. That is their explanation rather than a directly measured mechanism, but the direction of the two measures together is the relevant comparison: the harness is not buying success with extra policy calls.

Rigid objects, added latency, frontier models

The scope is bounded in three ways that qualify how far the result travels. The study covers rigid and articulated objects only; deformable-object manipulation is not addressed, and changing object geometry would add new challenges for scene representation and synchronization. Interaction-aware synchronization also costs time, particularly during grasp and release transitions that require hypothesis replay, interaction-consistency filtering, and contour matching, which limits responsiveness. And the harness relies on powerful multimodal foundation models for spatial reasoning and action generation; its effectiveness with smaller models remains unverified.

The last point is not incidental to the headline numbers. Every reported gain comes from one frozen GPT-6 Astra policy under one real camera configuration, so the result supports the claim that this policy's fine-manipulation capability was present but poorly observable — not a general claim that any policy would benefit equally.

Observability first, before retraining

The paper's most useful priority is a diagnostic one. When a strong multimodal policy fails at fine manipulation, the evidence here suggests checking whether the deciding spatial relationship is visible at all before assuming the policy lacks the capability. The clearest signal in the data is the split between tasks: where the baseline never succeeded (Tower of Hanoi, drawer), added views turned 0% into 100% and 66.67%; where it already succeeded sometimes (plug insertion, block stacking), added views still added 40.00 and 53.33 points. Both patterns point the same way, and the accompanying drop in policy queries argues against the gains being an artifact of more attempts.

The practical threshold the study implies is about what the real cameras can resolve near contact, not about task difficulty in general. SpatialHarness is aimed at cases where occlusion or viewpoint hides the geometry that decides success, and it delivers that without policy fine-tuning or new physical cameras — at the cost of maintaining a synchronized simulation, added transition latency, and dependence on a frontier model.

Sources used in this answer