Continual learning measured through skill recombination
The paper asks whether a self-evolving embodied agent can accumulate experience from simple tasks and reuse it on harder tasks it has never seen. Its answer rests on one design choice: continual learning is not measured by retaining old task performance, but by compositional generalization. An agent that has practiced basic tasks should recombine the skills it acquired to solve a novel advanced task, instead of learning that task from scratch.
This makes the benchmark's unit of evidence a task pair rather than a single episode. Basic tasks are short household operations built from a few atomic actions, such as picking up an object and putting it down. Advanced tasks chain more atomic actions drawn from two to four categories of basic tasks, so success on them depends on whether previously practiced skill elements can be assembled into a new sequence.
The authors argue this fills a gap: prior continual-learning work sits largely in non-embodied settings such as knowledge-intensive tasks or digital environments, while embodied benchmarks target perception, navigation, manipulation, or long-horizon planning rather than continual learning itself. EmbodiedCLBench is presented as the first embodied-simulation benchmark aimed at the continual-learning capability of self-evolving agents.
268 tasks, three evaluation settings, one success metric
The benchmark runs in TongSim, an indoor household simulator with 6 scenes, 51 object categories, and 7 atomic actions. It contains 268 tasks: 168 basic tasks across six household-activity categories and 100 advanced tasks. Each atomic action takes 3 steps, so a basic task with 2 atomic actions has a 6-step shortest path and an advanced task with 4 atomic actions has a 12-step shortest path.
Because the closed-source simulator cannot be run online for evaluation, the authors precompute a state-transition graph per task. Each node is an environment state paired with a first-person image and a set of candidate actions; each executable candidate is an edge that deterministically moves to the next state. Distractor paths branch off the shortest path and are executable but wrong. This lets evaluation run offline, in parallel, and reproducibly, while still requiring the agent to make sequential decisions.
Three settings separate what the agent is given. In zero-shot, it attempts basic and advanced tasks with no learning phase. In in-context learning (ICL), it first completes two related basic tasks per advanced task, and the full interaction trajectories stay in context during testing. In skill learning, it completes the same basic tasks but writes a concise Markdown skill summary, and only that summary is provided at test time. The reported metric is success rate: the share of tasks completed within the step budget, which is the shortest-path length plus 3 extra steps at test time and 6 during learning.
Four agent harnesses are paired with two backbone models through official APIs: OpenClaw, ClaudeCode, Codex, and Hermes, each with Qwen3.8-Flash and GPT5.6-Terra. That yields eight harness-model combinations, so every headline number below is a property of a specific pairing, not of embodied agents in general.
Advanced tasks collapse without a learning phase
The clearest baseline is how badly advanced tasks go when no experience is supplied. Across the eight combinations, zero-shot success on advanced tasks ranged from 3% to 12%. The same agents scored 42.9% to 58.9% on basic tasks in the same zero-shot condition.
That gap is the paper's motivating evidence. Basic tasks are within reach, but the compositional versions are not, even though they are built from the same atomic actions. The authors read this as showing that advanced tasks are hard to solve without leveraging prior experience, which is exactly the capability the benchmark is designed to probe.
The comparison also sets the reference point for everything that follows. Any improvement from a learning phase has to be judged against a 3% to 12% floor, and against the fact that the same agent already handles the component skills reasonably well in isolation.
Two basic tasks lift advanced-task success, unevenly
Adding a learning phase produced consistent improvement over zero-shot in every harness-model pairing, but the size of the gain varied widely. Under ICL, advanced-task success ranged from 16% to 52%; under skill learning, from 22% to 38%.
The spread matters more than the average. Codex with GPT5.6-Terra reached 52% under ICL, the best advanced-task result in the table, while the same harness with Qwen3.8-Flash reached 34%. Hermes with GPT5.6-Terra fell to 16% under ICL, below its own 34% under skill learning. OpenClaw with GPT5.6-Terra went the other way, from 39% under ICL to 22% under skill learning.
So neither learning method dominates across pairings. Retaining full trajectories helps some agents and hurts others relative to a compact written summary, which means the choice of how experience is stored interacts with the harness and model rather than being settled by the benchmark.
The authors also report that the benefit from basic-task experience remains stable as task complexity increases. That claim is about the persistence of the gain across harder compositions, not about the gain growing larger.
Extra learning samples do not translate into extra gains
The most consequential negative result concerns scaling the experience itself. The paper reports that improvement does not scale accordingly with the number of learning samples, and that current agents cannot extract meaningful information from additional experience.
This is a different question from whether learning helps at all. The positive result is that two related basic tasks beat zero-shot. The negative result is that adding more experience beyond that does not keep raising success. Read together, they locate the bottleneck in how agents consolidate and reuse experience, not in whether any experience is available.
The authors frame this as an open problem: how to enable agents to accumulate and exploit additional interactions remains unresolved. They also state that current performance is far from saturation, so the benchmark is not reporting a solved capability with diminishing returns, but an early capability with a specific failure to absorb more data.
Reading the results within their evaluation boundary
Several features of the setup bound what these numbers mean. Evaluation is offline and deterministic on precomputed state-transition graphs rather than live simulation, which is what makes parallel and reproducible scoring possible but also fixes the candidate action sets the agent chooses among. Task descriptions and action candidates were generated by Qwen3.7-Plus and manually verified, so the task text is model-produced and human-checked rather than authored from scratch.
The learning phase is also small by design: each advanced task is paired with two related basic tasks. Statements about how much experience helps therefore describe behavior at that scale, and the paper's own finding is that increasing the sample count did not produce proportional improvement.
Finally, all results come from eight harness-model combinations on one simulator. The paper's claim of consistent limitations and distinctive behaviors is a claim about that evaluated set, and the wide spread between pairings is itself evidence that results should be read per configuration rather than as a single embodied-agent score.
The priority is experience use, not experience collection
The paper's most useful contribution is a reframing of what to measure. If advanced-task success rises from a 3% to 12% zero-shot floor to as much as 52% after two basic tasks, then the binding constraint is not whether agents can be given experience. It is whether they can convert that experience into reusable skill structure, and the flat response to additional samples says they currently cannot.
That makes the benchmark's per-pairing spread the practical signal. A harness-model combination that scores well under ICL but poorly under skill learning is retaining trajectories without distilling them; one that improves under skill learning is compressing experience into something transferable. For readers judging self-evolving embodied agents, the informative comparison is between those two storage modes on the same pairing, and the informative threshold is whether added experience moves success at all.
The authors' stated direction follows from this: designing continual-learning methods that consolidate, generalize, and reuse experience across tasks, rather than methods that simply collect more of it.
Sources used in this answer
EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents
Introduces the first embodied-simulation benchmark for continual learning, measuring whether agents recombine skills from basic tasks to solve unseen advanced tasks; learning from two basic tasks beats zero-shot, but gains do not scale with more learning samples.