[CVPR/Arxiv 2025] AutoSkill: From Ephemeral Chats to Self-Evolving Digital Surrogates

TL;DR

AutoSkill is a breakthrough framework that allows AI agents to "learn" from you without retraining. By transforming your repeated requests—like "don't use jargon" or "format this as a script"—into structured, versioned SKILL.md files, it builds a persistent "Skill Bank." The next time you ask a question, the agent retrieves these skills to align perfectly with your established habits.

The Problem: The "Groundhog Day" of AI Interactions

Current LLMs suffer from a form of digital amnesia. Even with basic long-term memory (RAG), they treat your past conversations as a pile of text snippets. If you constantly ask an AI to write in a specific professional tone, you often have to remind it in every new session.

Prior Work Limitations:

• RAG (Memory-based): Good for facts (e.g., "My birthday is May 5th"), but poor at capturing how you want things done.
• Fine-tuning: Too slow, expensive, and a "black box"—you can't easily see or edit what the model has learned.

AutoSkill argues that we should treat interaction experience not as raw memory, but as Skill Formation.

Methodology: The Dual-Loop Engine

AutoSkill operates via two tightly coupled loops that separate "doing" from "learning."

1. The Skill Evolution Loop (The Teacher)

As you talk to the agent, a background process monitors your queries.

• Extraction: It identifies recurring patterns (e.g., a specific Python coding style).
• Management (The Judge): It decides if this is a new skill, an update to an old one, or just noise to be discarded.
• Versioned Merging: If you provide new feedback on an existing skill, it doesn't create a duplicate. It "bumps" the version (e.g., v0.1.2 to v0.1.3), refining the SKILL.md artifact.

2. Skill-Enhanced Response Loop (The Doer)

When a new query comes in, the agent:

1. Rewrites your query to be "retrieval-friendly."
2. Performs a Hybrid Search (combining semantic meaning and keyword matching) to find relevant skills.
3. Injects the executable instructions from the skills directly into the LLM’s prompt.

Fig 1: The dual-loop architecture of AutoSkill, separating online serving from background learning.

Real-World Evidence: Skills in the Wild

The authors tested AutoSkill on WildChat-1M, a dataset of real-world ChatGPT logs. They found that human-AI interactions naturally cluster into distinct "Skill Categories."

Fig 2: Programming and Writing dominate, but niche categories like "Psychological Counseling" also emerge.

Case Study: Continuous Refinement

Consider the professional_text_rewrite skill found in the study. In the SkillBank, this artifact reached Version 0.1.34. This means the system refined the "rules" for rewriting text 34 times based on user feedback, eventually creating a highly robust set of constraints (e.g., "Do not use contractions," "Preserve all numerical values").

Why This Matters: Transparency & Control

The most profound shift in AutoSkill is the use of SKILL.md. Because the skills are stored as Markdown files:

1. Human-Readable: You can open your Skill Bank and see exactly what the AI thinks your "skills" are.
2. Editable: If the AI misunderstood a preference, you can manually edit the Markdown file.
3. Portable: You could theoretically move your "Writing Style Skill" from one model (like GPT-4) to another (like Llama-3) effortlessly.

Conclusion and Future Outlook

AutoSkill represents a move toward Personal Digital Surrogates. Instead of a generic model, we get an agent that accumulates a "toolbox" tailored to our specific life and work. While currently training-free, future iterations could integrate these explicit skills back into model weights (distillation) or expand to multi-modal skills (e.g., learning how you like your images edited).

By externalizing behavior into artifacts, AutoSkill makes the "Lifelong" in "Lifelong Learning" both manageable and measurable.