Can dynamic scenario synthesis verify whether agent skills are reusable?

Existing tasks rarely exercise a skill

Self-evolving agents distill their own task-solving experience into skills: stored guidance with a condition describing when it applies and a principle describing what to do. Because that experience contains redundant exploration, mistakes, and task-specific details, a distilled skill can encode an incorrect procedure or knowledge that does not transfer. The paper defines the property worth checking as reusability: whether the guidance still helps when applied beyond the experience it came from.

The paper argues that neither obvious source of evidence answers this. Replaying the source task reproduces the targeted situation but only in its original context, so the evidence is relevant but not novel. Evaluating on other existing tasks is novel but those tasks were defined independently of the skill, so they need not expose the situation the skill addresses. The authors quantify the second problem: after 500 downstream tasks, depending on the access setting, 17–32% of skills were never used at all, and only 44–55% of skills were used in at least five distinct tasks. A skill that is never exercised leaves its reusability unobserved.

Synthesis turns verification into a controlled test

SkillSandbox shifts verification from selecting existing tasks to constructing a task and environment around the target skill on demand. The output is a verification scenario, defined as a task, a configured environment, and a success criterion, which is meant to be both skill-relevant (completing it requires addressing the situation the skill's condition describes) and novel (its context differs from the source experience).

Three components split the work. A Proposer uses the skill's condition, the source task, and the source trajectory to write a programmatic specification: which requirements must be reproduced so the condition actually arises during execution, and which details should be varied to make the context new. The specification is expressed only through construction operations the environment exposes, such as adding a distractor, removing an item, or altering a product option. A Builder then binds the specification's variables to concrete objects and values from the environment's data, compiles it into an executable construction program, and runs it. A validation loop, using the same model as the executor, discards scenarios that fail the Proposer's requirements or duplicate an earlier scenario, generating candidates until five valid scenarios exist per skill.

A skill is kept only if it pays off

For each synthesized scenario, a Verifier runs the same executor twice, with and without the skill, and compares the two trajectories at aligned steps where the pre-action observations match. Its score combines two signals: an executability indicator that is 1 when the action taken with the skill actually implements the skill's instruction, and a utility term measuring the difference in terminal success reward discounted by remaining action counts. Averaging over retained step pairs and then across scenarios gives a reusability score, and the verdict is Keep if that score is positive and Reject otherwise.

The three dimensions the paper names are therefore distinct questions rather than one score. Executability asks whether the principle can be carried out through admissible actions when the condition holds. Utility asks whether applying it raises task success relative to running without it. Efficiency asks whether it reaches completion in fewer actions. A skill that is incorrect, overly specific, or not actionable should show no benefit or degrade execution and be rejected.

Synthesized scenarios separate helpful from harmful skills

The first analysis holds the Verifier and rollout budget fixed and varies only where evidence comes from: five synthesized scenarios, five executions of the source task, or five random tasks, using Gemini 3.1 Flash-Lite on both benchmarks. Verdicts are scored against each skill's measured effect on held-out tasks, where a skill is labeled helpful if it raises success rate and harmful if it lowers it.

Synthesized scenarios produced the highest verdict F1 and the best-performing library on both benchmarks, with held-out success rate up to 15.7 percentage points above the alternatives. The failure of the alternatives is instructive. On ALFWorld, verdicts from source and random tasks barely separated helpful from harmful skills, and the libraries they retained fell below the No Skill baseline, with random-task verification even below the unverified library. The reported held-out success rates on ALFWorld run 56.4 for synthesized scenarios, 49.3 for source tasks, 50.7 for random tasks, and 44.3 for no skill; on WebShop they run 32.0, 33.3, and 52.9 across the compared settings. The authors' explanation is that source tasks expose the situation only in its original context while random tasks rarely expose it at all.

Verdict accuracy depends on the benchmark

Agreement between verdicts and held-out skill effects differs sharply by environment. On WebShop with Gemini 3.1 Flash as Proposer, precision was 96.0%, recall 82.8%, F1 88.9%, and specificity 94.4%. High precision and specificity mean retained skills usually stayed helpful on held-out tasks while harmful ones were filtered out; the 82.8% recall means most helpful skills were preserved, though some were wrongly rejected.

ALFWorld is weaker on the same measures: precision 39.1%, recall 81.8%, F1 52.9%, specificity 60.0%. The paper attributes the low precision largely to base rates, since only 23.9% of ALFWorld skills were helpful on held-out tasks. Recall stayed close to WebShop's, and helpful skills made up 39.1% of the verified library, higher than in the unverified library. The practical reading is that a Keep verdict is strong evidence of reusability on WebShop, while on ALFWorld the filter mainly improves the composition of the library rather than certifying individual skills.

Verification does not need a stronger model

Because the framework targets self-evolving agents, the authors test whether verification quality depends on model capability. They replace only the Proposer with GPT-5.6 Luna while sharing the skills, executor, Builder, and Verifier, then compare verdicts against the same held-out effects. On both benchmarks the executor's own model matched or exceeded the stronger Proposer on every reported metric and retained more helpful skills; the stronger Proposer rejected more helpful skills without rejecting more harmful ones.

The authors connect this to the division of labor: the language model is used for planning and rollouts, while construction and scoring are fixed procedures. That is the paper's explanation for why structured execution evidence reduces dependence on model strength, and it implies a self-evolving agent can screen its own skills without an external model stronger than itself.

Success gains can hide extra actions

Breaking reusability into its three dimensions shows why a single success number is not enough. On WebShop, agents took more actions on average with skills even when both runs succeeded; for Qwen3.5-27B and Gemini, skills improved task success but added inefficiency on tasks that would have succeeded anyway. The paper's conclusion is that success gains alone cannot assess reusability, because skill guidance can introduce additional actions that a success-only measure would not register.

ALFWorld shows a different split. Both Qwen models gained in success and saved actions, while Gemini showed high executability yet lower success and more actions, a case where the guidance can be followed but does not help. Across models, the reported executability shares of scenarios with at least one implementing action were 92.3, 95.5, and 95.5 on ALFWorld and 62.5, 76.6, and 81.1 on WebShop, so being executable is clearly not the same as being useful.

Where the Keep threshold earns its trust

The paper's most decision-relevant distinction is not whether synthesis beats static task pools, but where a Keep verdict can be trusted. On WebShop, a positive score carried 96.0% precision and 94.4% specificity against held-out effects, so a retained skill is likely to remain useful. On ALFWorld, with only 23.9% of skills helpful on held-out tasks, the same threshold mainly raised the share of helpful skills in the library from the unverified baseline to 39.1%, which is a filtering gain rather than a per-skill guarantee.

That asymmetry should set the priority for anyone adopting this approach: treat the verdict as a library-composition tool whose reliability must be checked against held-out effects in the target environment, and read the efficiency term alongside success, since WebShop skills raised success while adding actions on tasks that already succeeded. The paper's own scope is ALFWorld and WebShop with three models, so the balance between precision and recall reported here is the reference point to re-measure rather than assume elsewhere.

Sources used in this answer