Skill1: Unified Evolution of Selection, Utilization, and Distillation in LLM Agents

TL;DR

LLM agents often use "skill libraries" to store successful strategies, but usually, the retrieval of these skills and the creation of new ones are handled by separate, frozen modules. Skill1 changes this by training a single policy to handle the entire lifecycle—searching, re-ranking, executing, and distilling—using one shared reward signal. It achieves a 97.5% success rate on ALFWorld, proving that "co-evolution" beats modular design.

The Bottleneck: Disconnected Skill Lifecycles

Most skill-augmented agents follow a three-stage loop:

1. Selection: Finding the right strategy in the library.
2. Utilization: Executing the task based on that strategy.
3. Distillation: Turning the experience into a new, reusable skill.

The problem? Most existing researchers optimize these parts separately. If your retrieval module is frozen, it won't learn that certain skills are actually failing in practice. If your distillation module uses a separate "quality score," it might fill your library with "high-quality" text that doesn't actually help the agent win. This leads to optimization bottlenecks where one part of the system limits the others.

Methodology: One Policy, Two Frequencies

Skill1 addresses this by treating the entire process as a single sequence of actions optimized via Group Relative Policy Optimization (GRPO). The genius of the paper lies in how it assigns "credit" from a single 0 or 1 task-outcome reward to three different stages.

1. The Unified Workflow

The agent doesn't just act; it generates its own search query and its own re-ranking permutation before interacting with the environment. After finishing, it generates its own summary of the "lesson learned." All these tokens are part of the policy's output.

Note: The model generates the Query, the Rank, the Actions, and the distilled Skill Strategy in a continuous loop, all controlled by the same neural weights.

2. Decomposing the Reward

How do you know if a selection was good or if the distillation was useful based only on whether the final task succeeded? Skill1 uses a signal decomposition:

• Utilization Credit: The raw outcome $r(au)$ (Did we win?).
• Selection Credit (The Trend): Uses an Exponential Moving Average (EMA) of outcomes for each skill. The policy is rewarded (via NDCG loss) for ranking skills with higher long-term utility at the top. This is the "low-frequency" signal.
• Distillation Credit (The Variation): Does this new specific rollout perform better than the best existing skill in the library ($r(au)-{\hat{U}}_i$)? If yes, the "distillation" was truly novel and valuable. This is the "high-frequency" variation.

Experimental Mastery

The authors tested Skill1 on ALFWorld (household tasks) and WebShop (shopping).

SOTA Performance

Skill1 reached an average success rate of 97.5% on ALFWorld, notably outperforming RetroAgent and GiGPO. Its strength was most apparent in complex tasks like "Pick2" (composing multiple sub-procedures), where the ability to retrieve and follow precise, distilled strategies is a game-changer.

Above: Comparison of success rates across different task types. Skill1 shows significant gains in 'Heat' and 'Look' tasks where precise instruction following is required.

Convergence Dynamics

Training data showed a beautiful "sequential acceleration":

• First, Selection Precision climbs rapidly (the agent learns what it needs).
• Then, the Utilization and Distillation rates follow (the agent gets better at using and making the supply of high-quality tools).

Critical Analysis: Why This Works

The value of Skill1 is its Inductive Bias toward consistency. By rewarding re-ranking based on the trend of a skill's performance, it prevents the agent from being distracted by "lucky" successes. Simultaneously, the variation signal for distillation prevents the library from becoming a graveyard of redundant, mediocre strategies.

Limitations

1. Computational Cost: Training a library that grows up to 5,000 entries increases wall-clock time by roughly 1.3–1.7x compared to standard RL.
2. Context Window: As the library grows, the "Top-K" candidates injected into the prompt consume more tokens, which may eventually hit the limits of current Transformers.

Conclusion and Future Outlook

Skill1 demonstrates that the "management" of knowledge (selection and distillation) is just as trainable as the "execution" of knowledge. For practitioners, this means we should move away from hand-crafted retrieval logic and toward unified, differentiable agent architectures where the memory and the policy evolve together.

Academic Significance: This work positions skill libraries not as external databases, but as active, learned components of the policy's state space. It marks a shift from "LLM+Retrieval" to "Self-Evolving Cognitive Agents."