How risky is same-capability skill retrieval for agent task requirements?

Two skills can match a topic and still disagree

The paper studies a retrieval failure that topic-level relevance cannot see. Two skills may serve the same broad purpose yet rely on different resources, preconditions, or output interfaces. When a query names a specific requirement, only one of them may actually satisfy it, while the other remains a plausible topical match. The authors call this same-capability risk-exposure retrieval.

Their concrete illustration is a query that asks for a map and a distance dictionary as a two-item return value. A mapping skill that computes both but returns only the map matches the topic and violates the requested return interface. For a second query asking for the one-item interface, that same skill becomes the compatible choice. Compatibility therefore depends on the query, not on the skill's capability label alone.

This framing matters because a retrieved list can contain both the compatible skill and its query-inappropriate sibling. The benchmark is built to make that pair inspectable: each unit links a query requirement to a skill that meets it and to a same-capability skill that violates it, with evidence supporting the distinction.

Recall, HSR, and CleanHit answer different questions

The paper separates three outcomes that a single relevance score would blend. Recall@K asks whether the helpful skill appears in the top-K list. Harmful sibling rate (HSR@K) asks whether the marked conflicting sibling appears in that same list. CleanHit@K requires both conditions at once: the helpful skill is present and the marked sibling is absent.

These metrics have different denominators and meanings, and the paper keeps them apart. A high Recall@3 says nothing about whether the conflicting sibling also ranked highly. HSR@3 measures exposure of the annotated sibling, not the prevalence of all incompatible skills in a registry. CleanHit@3 is the strictest of the three because a helpful hit accompanied by the sibling does not count.

The distinction is central to reading the results. A retriever can look strong on Recall while still handing the downstream selection stage a list that contains the wrong-but-similar option.

890 units across three query protocols

SameCapRisk-Bench contains 890 units and 1,314 query cases across five mechanisms and twelve conflict types. The units are organized into three query protocols. Single-query contracts contain 466 units derived from source tasks, each with one query and a skill that conflicts with a concrete requirement. Source-role pairs contain 387 controlled units and 774 queries, in which two well-formed skills exchange helpful and conflicting roles when the requested evidence role changes. Output-interface pairs contain 37 author-derived units and 74 queries, where different observable output requirements reverse which of the same two skills fits.

The protocols also use different candidate pools. Source-role queries rank against a 774-candidate pool, while the other two protocols share a separate 7,615-candidate public pool. Because the protocols differ in task source, construction, and pool, the paper reports their scores separately rather than merging them into one number.

After reserving 55 single-query units and 12 interface pairs for development, 823 units and 1,235 queries remain for the held-out main evaluation. The label evidence varies by requirement type: the paper counts 160 mathematical or algorithmic certificates, 37 interface-contract checks, 37 tool/backend or numerical outcome contrasts, 387 static source-role comparisons, 241 source-contract or local-operation reviews, and 28 code-change cases.

Source-task contracts expose the sibling far more often

The sharpest result is the gap between protocols. On source-task contracts, four public skill retrievers expose conflicting siblings at HSR@3 of 0.737–0.881. On controlled source-role queries, the same measure falls to 0.099–0.12. The skills are held fixed in the source-role pairs; only the requested evidence role changes, which reverses which skill is helpful.

That contrast shows why a pooled score is not enough to characterize this behavior. Across the fixed mixture of 1,235 held-out queries, the four retrievers reach Recall@3 of 0.903–0.944 while HSR@3 remains 0.344–0.393. The pooled figure sits between the two protocol extremes and hides the fact that exposure is concentrated in the source-task setting.

The paper also reports where the conflicting sibling lands. For these four retrievers, it ranks first on 24.8–29.2% of source-task queries. Retrieving both skills is therefore not the same as ordering them correctly, and the ordering error is frequent enough to matter for what the next stage receives.

Ordering errors persist when both skills are retrieved

A separate comparison isolates the relative order of the two marked skills. All seven neural methods retrieve both on the same 543 main queries, including 309 single-query cases. On those single-query cases, helpful-first rates range from 0.521 to 0.696, so even the best point estimate leaves about 30% ordered the other way.

Reranking does not uniformly fix this. BGE reranking has a lower helpful-first rate than BGE dense ranking, 0.521 versus 0.563, on that subset. The subset covers 75.2% of single-query cases but only 24.3% of source-role cases, so it answers an ordering question when both candidates are available rather than replacing full-split retrieval scores.

The authors' interpretation is that recovering the helpful skill, ordering the two representatives, and returning a list without the marked conflict are three distinct problems. The benchmark is designed to report them separately rather than letting one stand in for another.

Longer reranker inputs improve first-place choice

The paper also examines what the reranker actually receives. Truncating long skills can remove the passages where the two skills' contracts differ, so the ranker may never see the text that distinguishes them. The main BGE dense setting uses 512 document tokens, and a 4,096-token run measures sensitivity to input length.

With the same BGE top-20 candidates, model weights, tokenizer, FP32 scoring, and tie rule, increasing the reranker's input budget from 512 to 4,096 tokens reduces source-task sibling-first errors by 8.27 percentage points. Only the batch size changes, from 16 to 1, to fit memory. The longer input leaves all 774 source-role lists unchanged.

The gain is narrower than it may first appear. On single-query contracts, CleanHit@3 changes by +0.0073, with 15 gains and 12 losses and a 95% group-resample interval of [−0.0164, 0.0303]. The output-interface change is −0.02 with an interval of [−0.08, 0.04]. Both intervals include zero. Better top-ranked selection can therefore coexist with little change in whether the conflicting sibling remains in the list, which is why the paper retains the original 512-token setting in the main comparison.

The benchmark measures exposure, not agent behavior

The conclusions concern the retrieval stage: whether a fixed candidate list contains a helpful skill and its marked, query-incompatible sibling. They do not establish which skill an agent will select or what happens during tool or skill execution. This scope follows from the labels, which rest on local query–skill contract evidence rather than whole-skill execution tests.

Generalization is limited by how the contrasts were constructed. A candidate-only diagnostic reached 59.66% accuracy on 466 single-query units using skill wording alone, which shows residual label cues that vary across conflict types. Independent human inter-annotator agreement has not yet been measured. The protocols also differ in source, pool, text length, and construction; source-role cases cover only ten declared groups, and output-interface cases form a small set of derived pairs.

A cross-family model audit offers some support for the labels. DeepSeek-V4-Pro and GLM-5.3 agreed on 124 of 128 original-order queries with κ = 0.94, and both models selected the recorded helpful skill for every query in both orders on 88 of 96 units. Review of the remaining units found construction defects, and five units were corrected in a separate data revision; the retrieval tables use the corrected texts.

Prioritize source-task contracts over pooled scores

For anyone evaluating a skill retriever, the paper's priority is clear: report source-task contracts separately from controlled source-role queries. The pooled HSR@3 of 0.344–0.393 across 1,235 held-out queries understates exposure where it is worst, at 0.737–0.881 on source-task contracts, and overstates it where role changes make the distinction easy, at 0.099–0.12.

The second priority is to treat CleanHit, not Recall, as the measure of a clean list. Increasing K from 3 to 20 for BGE-M3 raises Recall from 0.811 to 0.937, but HSR rises from 0.329 to 0.833. A longer list recovers helpful skills without necessarily making the list cleaner, so a retriever that looks better on recall may simply be exposing more conflicting siblings.

The reranker budget result adds a third, narrower lever. Expanding input from 512 to 4,096 tokens cut source-task sibling-first errors by 8.27 percentage points, but the CleanHit intervals included zero. Fixing the first-place choice and removing the sibling from the list are separate goals, and the paper's evidence supports the first more firmly than the second.

Sources used in this answer