How can onboard robot agents combine behavior trees with language model decisions?

The split between deciding and executing

The paper's answer to the title question is an architectural separation. An onboard Actor, driven by a language model, reasons about the next step, selects a skill, and uses the returned observation to inform its next decision in a reasoning-and-acting loop. The skills themselves are predefined behavior trees exposed as named tools, and the Actor cannot modify their implementations or guards.

That constraint is the point. Because skill logic is fixed and reviewable independently of the task-level policy, developers can test execution conditions before deployment. The model keeps flexibility over what to do next; the behavior tree keeps authority over whether and how a chosen action actually runs.

Behavior trees are the robotics-native structure the paper uses for this interface. Each physical skill's reactive sequence rechecks cancellation, elapsed time, and skill-specific guards on every tick, a periodic tree update. The skill body advances only if all guards pass on that tick, and a failed guard interrupts the branch and requests that the controller stop.

Guards stop motion before the next model call

The intervention tests target the responsiveness claim directly. Across nine interventions, commanded velocity reached zero a median 19 ms after local acceptance, while measured speed reflected the subsequent locomotion control and physical settling. Battery trials also included sitting down.

The mechanism behind that number is the tick-level check: guards run without further Actor calls, and the Actor cannot disable them through skill arguments. The paper reports that local checks stop ongoing navigation without waiting for another model decision, while the agent remains active to retry or report constraints. Context skills stay available so the Actor can gather information and report constraints even when physical actions are blocked.

Preconditions do a different job from guards. Before a physical skill starts, the framework validates arguments, resolves object references, and checks required prior observations, filtering available skills and rejecting ineligible calls. Guards then govern continuation, which matters because conditions such as localization can fail after navigation has already begun.

Success gains are largest for smaller models

On the combined 161-task text suite, BT-Agent matched or exceeded the strongest Act/ReAct baseline at every model size tested, with the largest gains at 4B and 9B: 10.1 and 5.0 percentage points. Transfer gains were positive across sizes and largest at 9B, at 4.8 points over ReAct, while the larger 4B gain came mainly from the primary suite.

The absolute numbers show why harness design matters more at small scale. At 4B, BT-Agent reached 51.6% success on the 161 tasks against 41.4% for ReAct and 35.6% for Act. At 9B, BT-Agent reached 54.7% against 49.7% for ReAct. At 27B the nominal comparison narrowed: ReAct reached 62.9% and BT-Agent 62.5%, with ReAct using lower median and P99 output.

The paper's traces suggest a reason. In 4B text traces of the safe task, ReAct visited the panels but submitted without pressing them, while BT-Agent completed the sequence and photographed the medication. In an inscription task, both identified the printed word, but only BT-Agent recorded it using the report skill. The authors read these cases as evidence for making required actions and procedures explicit in harness design, especially for smaller models.

Structured calls and failure feedback carry the gain

Matched ablations used a fresh control with the controller, skills, checks, and tasks fixed. Replacing structured tool calls with textual commands in action selection and history lowered success at every size, most at 4B and 9B, where median output rose 1.9× and 1.7× on nominal tasks.

Reducing failure feedback to a skill name and status lowered nominal success by 2 to 6 points across sizes. The paper attributes this to behavior-tree execution exposing failed guards or leaves, providing diagnostics as a by-product without another model call. Removing capture-specific guidance had little effect.

These ablations identify the action interface and informative failure feedback as the most useful components, and they show that the benefit is not uniform: smaller models are particularly sensitive to harness design, while simpler interfaces become more competitive at larger sizes.

Faults reveal differences nominal success hides

Under the injected faults, BT-Agent had the highest observed success at every size. The text backend applies repeatable faults to eligible skill calls, covering stale or missing pose, navigation that does not move or times out, route obstruction, incorrect viewpoints, presses that stop short, and capture analysis that fails to produce evidence. Some faults report failure explicitly; others report success despite incomplete execution, testing whether agents check outcomes beyond the returned status.

The 27B case is the clearest illustration of why fault testing belongs in deployment evaluation. There, the nominal tie with ReAct became a BT-Agent lead under faults. A paired 4B breakfast trace shows the behavioral difference: both methods succeeded without faults and received identical observations up to an injected route blockage, but BT-Agent changed viewpoint and completed, whereas ReAct retried the blocked approach and later repeated unsuccessful captures until exhausting its budget.

The watchdog is a separate, optional mechanism and should not be confused with the guards. It stops episodes after repeated unusable outputs, repeated equivalent queries or failed-skill outcomes, or reaching 16,384 output tokens or 128 model calls. It lowered P99 output at every size with minor impact on success, and token checks follow complete responses, allowing small overshoots.

Text trials are cheap but not a substitute

The multi-fidelity benchmark connects a 161-task text suite to physics simulation through shared task goals and skill interfaces. All tasks were evaluated in text, and 39 primary Habitat tasks were also evaluated in simulation. Text supported 10.9× higher evaluation throughput, while simulation revealed how perception and motion failures change task outcomes.

The agreement is real but partial. Method ordering agreed across backends, and tasks with higher success in text tended to have higher success in Habitat, with a Spearman correlation of 0.61 across 39 task means. Of 188 successful Habitat runs, 168 succeeded in text, but only 57.3% of text successes succeeded in Habitat. Symbolic success therefore does not establish completion with sensing and motion.

Paired traces show where the backends diverge. Both classes runs followed query, navigate, survey, capture but selected different cameras. On wallet/backpack, survey and capture disagreed about wallet visibility in the same image, causing repeated failed captures. On an absent-object task, one text run queried until timeout while its Habitat counterpart reported the unsuccessful lookup and submitted. Differences can arise in decisions and stopping behavior as well as in perception and motion.

What the onboard trials do and do not show

The physical demonstration used a 9B model on a quadruped across five task categories, with operator-adjudicated successes over three trials per task, totaling 11 of 15. One task, finding and capturing the round table with the Coke can, succeeded in 2 of 3 trials. Another, reading a clock, found and captured the screen and reported a time in all three trials, but small clock faces prevented correct visual readings.

The manipulation comparison uses a different execution path and should not be read as a behavior-tree result. In Robosuite, skills use blocking API calls rather than behavior-tree execution, and the comparison is between program generation and stepwise skill selection over six reference tasks plus eight sorting, stacking, and rearrangement extensions, with 20 repetitions per task at 4B, 9B, and 27B.

The architecture's own boundary is stated plainly: the Actor can sequence and parameterize available skills, but implementations and guards stay fixed at runtime, so new capabilities require skill authoring and platform-specific integration. Reliably conveying task-relevant evidence also remains open when observations are incomplete or ambiguous. The paper notes it has been submitted to IEEE for possible publication, so these results should be read as a preprint-stage report.

Where the architecture earns its place

The paper's most decision-relevant comparison is not the average gain but the pattern across conditions. BT-Agent's advantage over the strongest Act/ReAct baseline is largest at 4B and 9B, narrows to a nominal tie at 27B, and reappears under injected faults at 27B. The same design that helps most when the model is small and the environment misbehaves is the one whose overhead matters least when the model is large and conditions are nominal.

That pattern sets the practical threshold. If the deployment target is a small onboard model sharing compute with perception and control, the structured tool interface, enforced guards, and informative failure feedback are the components the ablations identify as carrying the gain. If the target is a larger model in nominal conditions, the simpler ReAct interface can match nominal success with lower output, and the case for the behavior-tree layer rests on fault handling and on stopping motion without another model decision.

The 19 ms median commanded-velocity stop and the fault-condition lead are the results that distinguish this architecture from a plain reasoning-and-acting loop. The 57.3% text-to-Habitat transfer rate is the reminder that neither number should be read as a guarantee of physical task completion.

Sources used in this answer