[ArXiv 2026] XSKILL: Bridging the Visual-Semantic Gap in Continual Learning Agents

TL;DR

XSKILL is a training-free, dual-stream framework that empowers multimodal agents to learn from their own successes and failures. By extracting Skills (structured workflows) and Experiences (context-specific tactics) grounded in visual observations, it solves the "forgetting" and "inefficiency" problems of current agents. Across major benchmarks, it boosts success rates by 2.58–6.71% and demonstrates remarkable zero-shot transfer across different AI models.

The Bottleneck: Stateless Agents in a Stateful World

Most multimodal large language models (MLLMs) today are "stateless"—they treat every new task as if it's the first time they've seen a tool. This leads to two major frustrations:

1. Inefficient Tool Use: Agents get stuck in loops, making the same syntax errors or using "Web Search" when a simple "Code Interpreter" script would suffice.
2. Inflexible Orchestration: They struggle to adapt general instructions to specific visual contexts (e.g., identifying a small object in a dark, inverted image).

While previous work tried using "trajectories" (text logs) as memory, text alone lacks the visual grounding necessary to understand why an action was taken. If an agent rotates an image, the text says "Image Rotated," but the reason (the image was upside down) is only in the pixels.

Methodology: The Dual-Stream Architecture

XSKILL introduces a closed-loop system for knowledge accumulation and inference, split into two distinct streams:

1. The Knowledge Accumulation (Phase I)

When the agent performs "rollouts" (trial runs), XSKILL doesn't just save the log. It uses a Critique Mechanism:

• Visually Grounded Summarization: Analyzes the image alongside the tool call to link visual evidence (e.g., low contrast) to the action (e.g., image enhancement).
• Cross-Rollout Critique: Compares successful vs. failed paths to isolate the "causal factors" of success.
• Hierarchical Consolidation: Merges similar insights to prevent the knowledge base from becoming a cluttered "data swamp."

2. The Task Execution (Phase II)

During inference, XSKILL doesn't just dump all memory into the prompt. It uses:

• Task Decomposition: Breaks the query into sub-tasks (e.g., "handling dark images") to retrieve specific experiences.
• Context-Aware Adaptation: An "Experience Rewriter" modifies global tips to match the current task's specifics, ensuring the advice is actionable and relevant.

Experimental Evidence

The authors tested XSKILL against SOTA baselines like Agent-KB and Dynamic CheatSheet using backbones like Gemini-2.5-Pro and GPT-5-mini.

Key Performance Gains

• Reliability (Average@4): Improved by up to 11.13 points on challenging benchmarks like TIR-Bench.
• Robustness: In the "Skill Only" setting, tool execution errors (like syntax or wrong tool names) dropped by nearly 50%.

Why it Works: Skills vs. Experiences

The ablation study provides a fascinating insight into agent "psychology":

• Skills act as the foundation: They reduce structural mistakes and prevent the agent from wasting its "step budget" on error recovery.
• Experiences act as the strategist: They shift tool usage toward more sophisticated methods. For example, in multimodal search, injecting experiences doubled the use of the "Code Interpreter" for precise data extraction rather than relying on fuzzy "Web Search."

Zero-Shot Generalization: Learning Once, Applying Everywhere

Perhaps the most impressive feat is XSKILL's transferability. Knowledge accumulated by a Gemini model could be effectively used by an o4-mini model. Furthermore, skills learned on "VisualToolBench" (general tool use) successfully transferred to "TIR-Bench" (integrated reasoning), outperforming baselines that were specifically tuned for those tasks.

Critical Analysis & Conclusion

XSKILL moves us closer to "Lifelong Learning Agents." By externalizing memory into human-readable Markdown (Skills) and JSON (Experiences), it also improves interpretability—human operators can audit and edit the "lessons" the agent has learned.

Limitations:

• The framework currently relies on a "larger" model (like Gemini-2.5-Pro) to act as the Knowledge Manager, which may increase latency in the accumulation phase.
• In weak open-source models (e.g., Qwen3-32B), the external knowledge occasionally "interfered" with native behaviors, highlighting that a baseline level of reasoning is required to benefit from XSKILL.

Final Takeaway: XSKILL proves that the path to smarter agents isn't just "more parameters," but a structured, visually-grounded way to remember the past.