Demystifying Agent Skills: Procedural Anchors, Retrieval Limits, and the Lifecycle of Reusable Experience

Core Overview

TL;DR

This paper studies when agent skills help, why they work, and where they fail by moving beyond end-to-end success rates. Through controlled experiments on Terminal-Bench, Terminal-Bench-Pro, SkillsBench, and paired trajectory analysis across Codex plus GPT-5.3-Codex and Gemini CLI plus Gemini-3.1-Pro-Preview, the authors isolate four variables: representation, outcome annotation, framework transfer, and retrieval difficulty. The central finding is that skills primarily work as procedural anchors, accounting for 65.7 percent of skill mechanisms, while explicit knowledge injection accounts for only 4.5 percent. Skills improve over workflow memory by 6.06 percentage points in matched paired comparisons, but retrieval becomes a separate failure surface: actual-use precision falls from 29.6 percent to 3.3 percent as skill pools grow from 5 to 100, even though downstream success changes only from 36.4 percent to 39.3 percent.

Positioning

This is not a new skill-generation system or a benchmark leaderboard paper. It is an empirical diagnostic study, closest in spirit to mechanistic analyses of agent memory: it asks which behavioral variables cause aggregate success changes. The paper sits after recent skill and workflow-memory work, including Anthropic-style agent skills, workflow memory, SkillsBench, and SWE-Skills-Bench, but its contribution is methodological and interpretive: it builds a paired contrastive taxonomy that makes skill use observable stage by stage. The background of the paper, shown in the pipeline comparison image below, frames skills not as a single artifact but as a sequence of decisions about representation, transfer, retrieval, invocation, and adaptation.

Problem and Motivation

The practical problem is subtle. Tool-using LLM agents often fail not because they lack high-level reasoning, but because they repeatedly rediscover operational details: environment setup, service lifecycle management, output formats, shell command discipline, debugging order, and verification checks. Recent systems therefore reuse traces of prior execution. The field has begun calling compact, standardized procedural artifacts “skills,” and these skills are increasingly presented as a better abstraction than raw trajectories or workflow memories because they can compress experience, standardize format, and potentially transfer across related tasks.

The research gap is not whether skills improve success rates; prior evaluations already suggest that they often do. The gap is that aggregate success does not explain what changes inside the execution trajectory. A skill-augmented run may succeed because the agent received more prior experience, because the experience was distilled into a better procedural representation, because failed trajectories were explicitly labeled as failures, because the skill transferred across frameworks, or because the agent happened to retrieve and invoke a compatible artifact. Existing evaluations collapse these distinct mechanisms into one scalar.

The paper identifies several concrete failure mechanisms that aggregate metrics hide. First, workflow memory preserves too much trace-level noise: failed branches, exploratory debugging, verbose command sequences, and context burden can cause timeouts or drift. Second, skills introduce a new abstraction-dependency failure surface: the agent must decide whether a skill applies, how to adapt it, which steps to follow, and when to abandon it. Third, retrieval quality is not the same as downstream success: confusable skill pools can degrade explicit selection while task success remains comparatively stable. Fourth, even when the correct skill is invoked, it may be insufficient for tasks requiring deep algorithmic reformulation or runtime verification. The authors therefore replace a black-box success question with a controlled pipeline question: how is experience represented, annotated, transferred, retrieved, invoked, and adapted?

Core Chapter: Mechanisms, Not Aggregates

Measurement: Paired Trajectories and a Human-Validated Taxonomy

The core unit of analysis is a paired triple. For the same task and setting, the paper compares three arms: Raw execution with no prior experience, Workflow Memory injection containing cleaned procedural traces, and Skill injection containing a distilled standardized skill built from the same source trajectories. This paired design matters because it holds the underlying experience fixed and changes only how that experience is packaged and made available. From the controlled experiments, the authors normalize 8,135 trial records into a shared manifest, 7,837 of which contain agent transcripts. They then open-code 240 sampled trajectories and retain 238 valid unique labels, which are consolidated into three high-level categories and 12 skill-use modes.

The main paired sample contains 528 triples, covering SkillsBench with 144 triples, Terminal-Bench 2.0 with 186 triples, and Terminal-Bench-Pro with 198 triples. Each triple yields three arm-level labels, giving 1,584 arm-level mode assignments. The high-level categories are intentionally interpretable. SC1 captures successful procedural anchoring, where either the agent succeeds autonomously or prior experience provides useful guidance. SC2 captures execution-layer and verification failures, such as environment setup, output formatting, service management, shell execution, algorithmic implementation, and runtime validation. SC3 captures invocation, applicability, and boundary failures, where guidance is present but misused, over-applied, ignored, or constrained by external limits.

The taxonomy is not merely an LLM summarization of failures. Table 3 reports a two-stage human validation. In the first stage, for each of the 238 valid raw labels, a human annotator inspected three supporting trajectories, yielding 714 trajectory-label checks, and all labels were confirmed as grounded in recorded agent behavior. In the second stage, a human independently mapped all 238 raw labels to the 12 canonical modes, achieving 95.8 percent exact agreement with the LLM aggregation and a Cohen kappa score of 0.952. This human check is central to the credibility of the paper: the qualitative mechanism claims, such as procedural anchoring versus knowledge injection, would otherwise depend heavily on a single LLM judge’s interpretation.

Representation: Skills Are More Than Shorter Traces

RQ1 asks whether the same prior experience behaves differently when represented as workflow memory or distilled as a skill. This is the paper’s most important controlled comparison because both Workflow Memory and Skill arms are constructed from the same selected trajectories. The central results support a modest but meaningful conclusion: skills are not simply useful because they expose the agent to more past experience; the representation changes behavior.

Table 9 reports oracle-status success rates over the 528 paired triples. Raw execution achieves 312 successes out of 528, or 59.1 percent. Workflow memory achieves 295 successes out of 528, or 55.9 percent. Skills achieve 327 successes out of 528, or 61.9 percent. The strongest aggregate effect is therefore not skill versus raw, but skill versus workflow memory. Table 10 makes this explicit through paired success-rate deltas and 1,000-iteration bootstrap confidence intervals.

Table 10 reports paired success-rate deltas across the three execution arms:

The table shows that the only positive lower bound belongs to Skill versus Workflow memory. In the interval notation, both bounds are percentage points. If the comparison were skill versus raw, the confidence interval still crosses zero, so the most robust conclusion is not that skills dominate raw execution under all settings, but that skills outperform workflow memory when the underlying experience is held fixed. That distinction is important. The paper’s argument is not “memory always helps,” but “distillation helps because it changes the action structure of reuse.”

The mechanism distribution explains why. Under the paired-triple taxonomy, procedural_anchor accounts for 65.7 percent of skill mechanisms, whereas knowledge_injection accounts for only 4.5 percent. A procedural anchor is a usable ordering, checklist, tool sequence, or verification plan. Knowledge injection supplies concrete facts that the agent otherwise lacked. The observed dominance of anchoring means that skills usually stabilize how the agent acts: which setup steps to run, which tools to call, which checks to perform, and which recurring pitfalls to avoid. They rarely function as external encyclopedias of missing facts.

Table 11, selected here as an excerpt of the mode distribution, makes the behavioral decomposition concrete. Percentages are computed within each arm over the same paired-triple sample.

Table 11 reports selected taxonomy mode percentages across arms:

The interpretation has two halves. First, skills sharply reduce execution-layer failures. Environment infrastructure failures fall from 5.3 percent in raw execution to 0.2 percent with skills. Output-format mismatches fall from 7.4 percent to 3.2 percent. Background service lifecycle failures fall from 2.7 percent to 0.8 percent. These are precisely the operational fragilities that procedural anchors address. Second, skills do not solve reasoning-bound failures. Algorithmic logic error remains substantial at 8.3 percent for raw, 11.0 percent for workflow memory, and 7.4 percent for skills. Static verification without runtime remains near 12 percent across all arms. A skill can tell the agent to verify latency, run tests, or avoid a bad serialization order, but it cannot automatically replace a wrong algorithm or force oracle-aligned runtime validation.

The same table also exposes the new failure surface created by abstraction. Skill-guidance misapplied or ignored appears in 10.0 percent of skill-arm cases, compared with only 0.8 percent in raw execution and 0.4 percent in workflow memory. This is the conceptual inverse of workflow memory’s weakness. Workflow memory burdens the agent with process noise: timeout budget exhaustion rises to 10.6 percent in the workflow arm, compared with 1.7 percent in raw and 4.4 percent in skills. Skills reduce that burden through compression, but they introduce an applicability-judgment burden. A skill is not self-executing; it must be noticed, interpreted, adapted, and abandoned when its assumptions fail.

The paper further tests whether the skill advantage could be explained by compact procedural text alone. Appendix Table 12 adds two lightweight baselines on the same 26 selected Terminal-Bench-2 tasks used in the Gemini comparison. Raw execution reaches 50.0 percent success. An instruction-derived short plan reaches 47.7 percent. A workflow-derived test-first template reaches 59.2 percent. Workflow Memory reaches 62.3 percent. Skills reach 79.2 percent. If the benefit were merely “a short hint helps,” the short plan and test-first baselines should have closed most of the gap. They do not. The result strengthens the representation argument: useful distillation appears to preserve the right operational procedure while discarding incidental noise, but the paper does not claim that every compact procedural hint becomes a skill.

There is also an effectiveness-efficiency trade-off. Appendix Table 13 reports token and success statistics on an 83-task matched intersection. Workflow memory is the most token-efficient representation, with an average total of 426.2 thousand tokens per task, compared with 555.7 thousand for raw trajectories and 521.5 thousand for skills. Its success rate is 64.8 percent versus 64.1 percent for raw. Skills are not always cheaper than workflow memory; they use 95.3 thousand more total tokens than workflow memory on the matched intersection. However, skills achieve 69.6 percent success, improving over raw by 5.5 percentage points and over workflow memory by 4.8 percentage points. The trade-off is clear: workflow memory may be cheaper, while skills buy execution robustness at additional context cost.

Retrieval: A Separate Bottleneck from Selection to Use

RQ4 asks whether skills remain usable when retrieved from larger and more confusable pools. This is where the paper’s findings are perhaps most practically important for self-evolving agents. The authors define retrieval over SkillsBench’s ground-truth task-skill annotations. For a record, let $G_t$ be the gold skill set attached to the task by the benchmark, and let ${\hat{G}}_i$ be the distinct skills predicted by the offline retriever, explicitly selected by the agent, or actually inspected or invoked during execution. The metric definitions are simple:

In this expression, $P_i$ is the precision of record $i$, $|{\hat{G}}_i|$ is the number of distinct skills predicted or accessed, and $|{\hat{G}}_i\cap G_t|$ is the number of predicted or accessed skills that exactly match a canonical gold identifier. The role of the formula is to measure whether an agent’s operationalized skill use is concentrated on annotated ground truth. If $G_t$ has at least one item, but every accessed skill is a distractor, precision is zero. If the accessed set contains no skills, the paper sets $P_i=0$, which treats non-use as a failure to operationalize available knowledge.

The recall counterpart is:

Here $R_i$ is the fraction of the gold skill set that the agent considered, and $|G_t|$ is the number of annotated gold skills for the task. Unlike precision, recall is not directly comparable across records when $G_t$ is small, and it does not require the predicted set to be exact. A high recall with low precision can mean that the agent saw the correct skill among many unrelated or related distractors. This is precisely the pattern the paper observes in Arm 3 real execution: recall at $k=100$ remains between 54.3 percent and 73.6 percent, while precision collapses to between 0.7 percent and 8.1 percent. That combination supports the paper’s claim that exact ground-truth invocation is neither sufficient nor strictly necessary for task success. The agent may still extract useful procedural support from a nearby but non-gold skill.

Table 4 provides the full pool-size trend behind this conclusion.

Table 4 reports precision and downstream success across pool composition and size:

The table contains three different kinds of measurements, and the paper is careful to separate them. Arm 1 is an offline embedding-retrieval diagnostic using Qwen3-Embedding-0.6B. Arm 2 is explicit offline agent selection without downstream execution. Arm 3 is real execution with the complete pool available, followed by parsing the skills actually inspected or invoked. The outputs of the offline arms are not passed into the execution experiment. This is a crucial methodological choice because it prevents the authors from treating “selected skill” as a deterministic input to “successful execution.”

The pattern shows that retrieval is a separate bottleneck with a weak coupling to downstream success. In Arm 3, actual-use precision collapses across all distractor regimes as the pool grows: random pools fall from 25.9 percent to 4.4 percent, similar pools from 34.5 percent to 3.7 percent, and dissimilar pools from 28.6 percent to 1.7 percent. Yet downstream success remains comparatively flat: random pools move from 31.8 percent to 41.9 percent, similar pools from 41.7 percent to 39.6 percent, and dissimilar pools from 35.7 percent to 36.4 percent. The abstract in the paper summarizes this as actual-use precision falling from 29.6 percent to 3.3 percent while downstream success changes only from 36.4 percent to 39.3 percent across the two reported pairings. This is a strong warning for practitioners: if one evaluates only end-to-end success, retrieval degradation in large skill libraries may be invisible. If one evaluates only offline retrieval accuracy, one may overstate the importance of exact top-level selection. The reality is that execution depends on noticing, adapting, and operationalizing guidance, not merely on retrieving the annotated ground-truth skill.

The composition analysis adds another layer. Semantic confusability is a larger stressor than raw pool size for offline identification. In Arm 1, top-1 precision on similar pools falls from 70.5 percent at $k=5$ to 53.4 percent at $k=100$, while random pools fall from 97.7 percent to 84.1 percent and dissimilar pools remain high, from 96.6 percent to 93.2 percent. In explicit agent selection, similar pools also cause difficulty: Gemini selection precision is 54.3 percent at $k=5$ and 55.4 percent at $k=100$, while Codex selection precision drops from 51.9 percent to 31.9 percent. However, Arm 2 recall at $k=100$ remains 70.5 percent to 85.2 percent across reported conditions. Agents often include the ground-truth skill together with distractors rather than failing to consider it at all. This reinforces the paper’s lifecycle view: useful skill exists in the pool, but existence is not the same as correct use.

Annotation and Transfer: Outcome Labels and Framework Portability

RQ2 isolates outcome annotations. The authors construct skills from the same trajectory pools with and without success or failure labels visible during skill creation. Figure 5 and Appendix Table 16 show that outcome labels matter most when failed trajectories enter the pool. For example, on Gemini Terminal-Bench-2 at the 3s2f mixture, normal skill creation reaches 0.7462 success, whereas the no-hint variant reaches 0.4000. The same directional pattern is reported across completed Gemini Terminal-Bench-2 and SkillsBench ratios once failures are included. This is an important refinement of the procedural-anchor story: the distilled guidance is often improved not simply by more procedural content, but by knowing which observed branches are reliable and which are failure signals. Failed trajectories contain exploitable warnings, but only if the creation process can distinguish them.

RQ3 asks whether procedural guidance transfers across frameworks. The design constructs workflow memories and skills in the Codex setting, then evaluates them in Gemini CLI with Gemini-3.1-Pro-Preview. This isolates framework portability: source experience is fixed, while prompting style, tool interface, and execution loop change. Figure 4 visualizes transferred Skill and Workflow Memory against the target framework’s raw baseline. The provided paper presents this as a controlled transfer study, but the main text excerpt does not transcribe a compact numeric paired-delta table for cross-framework transfer in the same way it does for representation and retrieval. Therefore, the safe reading is that transfer is treated as a boundary condition rather than the paper’s headline quantitative contribution. The conceptual point is still valuable: if procedural knowledge is coupled to a scaffold, distillation should be evaluated by whether it survives framework changes, not merely by textual similarity.

Experiments and Evidence

Main Evidence: Procedural Anchoring over Workflow Memory

The strongest quantitative evidence supports the claim that skill value comes from representation, not simply from adding experience. The paired analysis in Table 10 shows that Skill versus Workflow memory has a mean delta of +6.06 percentage points, with a bootstrap interval of [+0.76, +11.36] percentage points. Since both arms are built from the same source trajectories, this difference cannot be explained by “more prior experience.” It must come from how that experience is packaged. The taxonomy then attributes the mechanism to procedural anchoring: 65.7 percent of skill mechanisms are procedural anchors, versus 4.5 percent knowledge injection.

This evidence is stronger than an end-to-end success comparison because it is paired, matched, and behaviorally decomposed. Raw, workflow memory, and skill trajectories are compared for the same task and, where applicable, the same trajectory-mixture setting. Table 9 shows success rates for raw, workflow memory, and skill at 59.1, 55.9, and 61.9 percent respectively, but the more informative statistic is the paired delta table because it uses the same task-setting comparisons rather than an unpaired average.

The evidence quality is high for the representation claim but modest in magnitude. The point estimate of +6.06 points is not huge, and the confidence interval lower bound is only +0.76 points. The paper does not claim that skills always win; it claims that when compared against direct workflow memory, skills often produce a better operational form of experience reuse. That is a defensible and useful conclusion.

Disconfirming Alternative Explanations

Several controls help exclude simpler explanations. The lightweight compact-text baselines in Table 12 rule out the possibility that any short procedural hint is enough. On the same 26 Terminal-Bench-2 tasks, short plans achieve 47.7 percent and test-first templates achieve 59.2 percent, while skills achieve 79.2 percent. This is not the expected pattern if compactness alone caused the gain.

The outcome-annotation ablation also refines the claim. If skills were purely procedural, labels might not matter much. But Table 16 and Figure 5 show that when failed trajectories are present, outcome labels can matter a great deal. For Gemini Terminal-Bench-2 3s2f, normal creation reaches 0.7462 and no-hint reaches 0.4000. This suggests that failure labels help the skill creator decide what to extract, what to generalize, and what to present as a warning. The paper therefore does not say “labels always improve skills”; it says that labels become especially useful when the source pool contains noisy or unsuccessful trajectories.

The cross-framework study adds an important portability boundary, although the provided text emphasizes the design and figure more than compact numeric deltas. The conceptual message is that a skill created in one scaffold may not transfer automatically to another scaffold. The paper treats transfer as a lifecycle stage rather than a single artifact property.

Retrieval Evidence and the Decoupling of Precision and Success

The retrieval experiments are the paper’s second major empirical contribution. Table 4 shows a striking decoupling: Arm 3 actual-use precision collapses from tens of percent to single digits as pool size increases, but success remains comparatively stable. This prevents a common evaluation error: treating top-1 retrieval precision as a proxy for end-to-end agent capability.

The mechanism analysis explains why. In Arm 3, recall remains high: 54.3 percent to 73.6 percent at $k=100$. Agents often access multiple candidates, including the ground-truth skill, but do not restrict use to the annotated skill. The paper interprets this carefully: exact ground-truth invocation is neither sufficient nor strictly necessary. A retrieved non-gold skill may still contain transferable procedural guidance, while a gold skill may be invoked superficially or in the wrong context. This is a sophisticated result because it separates three levels: identification, invocation, and operational usefulness.

The offline diagnostics show where the bottleneck is harder. In Arm 1 and Arm 2, precision decreases as pool size grows, especially for similar distractor pools. But in Arm 3, success is insensitive to this precision collapse. The paper’s design makes this interpretable because the offline diagnostics are independent of execution. The authors do not feed selected skills into the downstream run. Therefore, they are not claiming a causal chain from offline selection to execution. They are comparing complementary aspects of skill use: can a retriever identify the right artifact, can an agent explicitly select it, and does the agent operationally use it during execution?

Failure Modes and Limitations of the Evidence

The most important negative finding is not that skills fail to help, but that they introduce and preserve distinct failure modes. The SC counts in the text show the trade-off: skill arms assign 326 of 528 trajectories to SC1 successful-procedure class, compared with 294 for workflow memory; skill arms reduce SC2 execution-layer failures to 124, compared with 197 raw and 176 workflow; but skill arms increase SC3 invocation or boundary failures to 78, compared with 19 raw. This is the central caution of the paper. Distillation helps operational robustness, but abstraction creates applicability risk.

The paired example in Appendix A.3 makes the mechanism concrete. In a React performance-debugging task under the 1s4f mixture, raw execution bypasses an external-service check and fails; workflow memory repairs one serialization problem but leaves a dependent profile request serialized and still fails; skill guidance tells the agent to start promises early and await late, producing a parallelized checkout path that passes all 11 verifier tests. The skill is not providing a new fact about React or HTTP. It is providing a reusable operational pattern: start independent work early, await dependencies as soon as possible, and preserve latency budgets. That is procedural anchoring in action.

The evidence also has limits. The experiments focus on terminal and tool-using benchmarks with test-based verification. These tasks are well suited to procedural reuse, but they may not represent long-horizon web agents, open-ended collaboration, or highly semantic domains. The model set is also limited to a small number of agent-model pairings, and the paper explicitly notes that RQ4 uses Codex with GPT-5.4 because GPT-5.3-Codex was unavailable under the same evaluation access. Therefore, RQ4 numbers are interpreted only within their own comparisons. The taxonomy is induced from 238 valid labels, roughly a 3 percent open-coded sample of the normalized record set, and full paired labeling remains LLM-assisted rather than exhaustively human-coded. These caveats do not overturn the main mechanism claims, but they do define their scope.

Deep Insights and Conclusion

Why This Matters

The paper’s lasting contribution is conceptual: skill utility is a lifecycle property, not a property of a text file. A skill helps only when noisy experience is distilled to the right abstraction, transferred or retrieved for a compatible context, invoked by the agent, and adapted during execution. The phrase “procedural anchor” captures the useful part of this lifecycle: a skill reduces uncertainty in action ordering, tool usage, verification, and operational constraints. The phrase “invocation failure” captures the missing part: the agent still has to decide whether the anchor applies to the present task.

This reframes future skill systems. If skills work mainly through procedural anchoring, then automatic skill generation should optimize for reusable operational structure: steps, preconditions, failure warnings, and verification plans. The skill-creator prompts in Appendix B already point this way, asking for Use This Skill When, Preconditions, Steps, Common Failure Modes, If A Failure Happens, and Verify sections. If retrieval fails as pools grow, then skill libraries need more than better embedding models; they need indexing by applicability, conflict, and operational scope. If a correct skill is not sufficient, then execution-time monitoring should measure not only whether a skill was selected, but whether it was adapted, verified, or merely quoted.

The Real Boundary of Skills

The most important boundary is between procedural fragility and conceptual failure. Skills are good at reducing operational fragility: missing dependencies, bad setup sequences, output formatting mistakes, service lifecycle errors, and shell corruption. They are weak at fixing algorithmic errors, replacing missing verification, or resolving deep problem reformulation. Table 11 shows that algorithmic logic error and static verification without runtime persist across arms. A skill can tell an agent to run tests, but it cannot make the tests semantically equivalent to the intended behavior if the agent still misunderstands the task.

The second boundary is retrieval realism. In controlled small pools, the right skill may be almost trivially available. In larger, semantically confusable pools, agents inspect and invoke many artifacts. This creates a regime where high recall and low precision can coexist. For engineering practice, this means that “did the agent choose the ground-truth skill” is a poor single metric. It is too strict if related skills provide useful guidance, and too lenient if the agent superficially invokes a correct skill while failing to adapt it.

Final Assessment

This is a strong diagnostic paper because it converts a vague success-rate story into a mechanism-level account. The numbers are not presented as a leaderboard win; they are presented as constraints on interpretation. Skills beat workflow memory by 6.06 points in matched analysis, but their advantage comes from procedural anchoring, not knowledge injection. They reduce operational failures, but they can increase misapplication failures. They remain useful even when actual-use precision collapses, but this stability is not evidence that retrieval is irrelevant; it is evidence that downstream execution is driven by a different variable than offline skill identification.

The practical lesson is uncomfortable but valuable: self-evolving agents do not become better merely by accumulating more memories. They need a skill lifecycle: distill experience at the right abstraction, preserve outcome signals when failures are present, transfer artifacts across frameworks, retrieve compatible guidance, and monitor whether the agent actually operationalizes it. The paper does not provide a complete solution to that lifecycle. It provides a taxonomy and evidence base for what must be measured next.