SkillOS: Driving Agent Self-Evolution through Reinforcement Learning-Based Skill Curation

TL;DR

SkillOS is a novel RL training recipe that transforms LLM agents from transient solvers into self-evolving systems. By training a dedicated Skill Curator to manage a Markdown-based repository, the framework enables agents to distill, refine, and reuse experience. It achieves SOTA-level efficiency and effectiveness by evaluating memory operations based on their impact on future related tasks.

Problem & Motivation: The "One-Off" Trap

Despite the reasoning prowess of modern LLMs, most agents suffer from "amnesia." Every task is a fresh start. While recent works have introduced "procedural memory" or "skill banks," they usually rely on:

1. Fixed Heuristics: Rigid rules (e.g., "always store a success") that don't adapt to the executor's specific failures.
2. Short-Horizon Signals: Curation is often judged by how well it summarizes a past task, not how much it helps a future one.

The authors of SkillOS argue that for an agent to truly evolve, curation must be treated as a long-horizon decision-making problem where the reward is the executor’s future success.

Methodology: The Skill Curator and Grouped Rewards

SkillOS adopts a modular multi-agent design, separating the "brain" into two parts:

• Agent Executor (Frozen): Retrieves skills via BM25 and acts in the environment.
• Skill Curator (Trainable): Observes past trajectories and manages the SkillRepo using insert_skill, update_skill, and delete_skill functions.

The Secret Sauce: Grouped Task Streams

To provide a valid learning signal, the researchers group tasks by latent attributes (topic, pitfalls, etc.). Within a group, the skills generated from Task A are immediately tested on Tasks B and C. This exposure allows RL to reward the curator only if its "edits" to the repo actually make the executor faster or more accurate later.

Figure 1: The SkillOS loop where raw experience is distilled into a Markdown-formatted SkillRepo.

Composite Reward Function

The training uses GRPO (Grouped Reward Policy Optimization) with a four-part reward:

• Task Outcome: Did the executor succeed on subsequent tasks?
• Function Call Validity: Were the curation commands syntactically correct?
• Content Quality: Measured by an external judge for semantic value.
• Compression: Rewarding conciseness to prevent the repo from becoming a bloated transcript log.

Experiments & Results: Efficiency Meets Generalization

SkillOS was tested on ALFWorld (embodied AI), WebShop (web navigation), and high-level reasoning (AIME).

Key Findings:

• Superiority Over Scale: A trained 8B Qwen model acting as a curator outperformed the much larger Gemini-2.5-Pro when the latter was used in a zero-shot prompting mode.
• Efficiency Gains: Agents became "smarter, not just harder working," completing tasks with significantly fewer environment steps.
• Zero-Shot Backbone Transfer: A curator trained using a Qwen executor still improved the performance of a Gemini executor, suggesting that "good curation" is a universal trait.

Table 1: SkillOS performance across different executors. Note the steady rise in Success Rate (SR) and decrease in Steps.

Deep Insight: The Evolution of Strategy

Analysis of the SkillRepo dynamics revealed a fascinating shift. Early in training, the curator mostly "blindly inserts" items. As RL progresses, it shifts toward updating and refining. The skills themselves evolve from simple "how-to" guides into sophisticated meta-strategies including failure-handling logic and conditional planning.

Figure: The shift from 'Insert' dominance to 'Update' dominance as the curator learns to manage existing knowledge.

Critical Analysis & Conclusion

Takeaway

SkillOS proves that we don't need to fine-tune the "worker" (executor) if we can train a "manager" (curator) to provide the worker with better tools. This modularity is a massive win for production systems where re-training large 70B+ models is prohibitively expensive.

Limitations

• Retrieval Bottleneck: The reliance on BM25 limits the system to keyword matches. Future iterations should likely move toward learned, dense retrieval.
• Flat Structure: Currently, skills are flat Markdown files. Real-world expertise is often hierarchical (Skill A calls Sub-skill B).

Looking Ahead

The next frontier for SkillOS lies in shared memory—allowing multiple agents to curate a common repository, creating a "departmental" intelligence that survives even if individual agent instances are reset.