SkillSynth: Scaling Terminal Agents via Scenario-Mediated Skill Graphs

TL;DR

Tencent's Hunyuan team presents SkillSynth, a pipeline that transforms atomic command-line skills into complex, multi-step terminal tasks. By organizing skills into a "Scenario-Mediated Skill Graph," the framework generates 3,500+ diverse, verified tasks. Training on this data enabled a 32B model to outperform a 480B giant, proving that trajectory diversity outweighs raw dataset size.

Background: The Diversity Bottleneck

Terminal agents (LLMs acting via CLI) struggle not because of a lack of parameters, but a lack of "experience." Most training data comes from GitHub issues or narrow software puzzles. This leads to agents that are brittle when facing "long-tail" scenarios—like fixing an audio driver or configuring a 3D simulation.

The authors argue that a terminal trajectory is defined by two axes: where you are (Scenario) and what you do (Skill). Current synthesis methods fail because they don't explicitly control the combination of these two, leading to redundant, easy-to-solve data.

Methodology: Building the Skill Graph

SkillSynth's breakthrough is treating terminal workflows as traversals on a graph.

1. Graph Construction

The team collected skills from ClawHub and GitHub, then used LLMs to "infer" the preconditions and postconditions for each.

• Nodes: Scenarios (e.g., "Python project with failing tests").
• Edges: Skills (e.g., run pytest and capture output).

By aligning the postcondition of one skill with the precondition of another, they created a massive Scenario-Mediated Skill Graph featuring 82,073 scenarios and 57,214 transitions.

2. Inverse-Frequency Sampling

To avoid "hub" nodes (standard states that would lead to repetitive data), the authors employed Inverse-Frequency Path Sampling. This forces the sampler to explore rare skills and scenarios, ensuring the training set covers the widest possible state-action space.

3. The Multi-Agent Harness

Generating a task isn't just writing a prompt. SkillSynth uses a "Planner-Constructor" architecture:

1. Planner: Converts a graph path into a high-level objectives.
2. Constructor: Builds the Docker environment, file system snapshots, and evaluation scripts.
3. Verify-then-Repair: An Oracle tries to solve the task. If it fails, the agent uses the error logs to "repair" the task instance.

Experiments & Results

The results validate the "Quality over Quantity" hypothesis. Tasks generated via SkillSynth were significantly harder, with a high failure rate for even advanced models like Claude 4.5.

Key Performance Highlights:

• Data Efficiency: Models trained on SkillSynth trajectories outperformed those trained on randomly composed multi-skill tasks by 3.8 points on Terminal-Bench 2.0.
• Scaling Victory: Qwen3-32B fine-tuned with SkillSynth (+SS) achieved 29.6% accuracy on Terminal-Bench 2.0, remarkably beating the Qwen 3 Coder 480B (23.9%).

Critical Insights: Why do Agents Fail?

The paper includes a fascinating error analysis of failed trajectories. The primary culprit? Partial Implementation (42.2%). Agents often run a self-test, see it pass, and confidently quit—even if they ignored half of the user's instructions. This suggests that future research should focus on "Instruction Alignment" and curbing "Self-Test Over-trust."

Summary

SkillSynth represents a shift from breadth-first task generation to structure-first trajectory synthesis. By grounding tasks in a graph of real-world workflows, it provides a repeatable, scalable blueprint for moving beyond simple LLMs toward truly autonomous terminal agents.

Limitations: Currently, the graph samples "chains" (sequences). The authors note that sampling "subgraphs" (parallel tasks) would further increase complexity and realism.