What do long-horizon benchmarks actually miss?
Long-horizon skill tasks typically measure whether an agent can execute a sequence of skills to reach a goal, but they rarely check whether the agent learned to choose the right skill in the first place. A 2026 paper identifies this as a structural gap: the skill-selection decision—which skill file to read or which skill to invoke—is made by the policy itself mid-episode, yet no existing training signal directly teaches it [1]. In other words, the benchmark rewards the outcome (did the task succeed?) but not the choice (did the agent pick the correct skill among many candidates?).
The paper names the underlying problem 'selector credit starvation': under a standard outcome-rewarded reinforcement learning setup, the few tokens that name the chosen skill carry a vanishingly small share of the loss, and the credit they do inherit becomes increasingly wrong-signed as trajectories lengthen [1]. Concretely, a correct skill choice is punished whenever the execution after it fails, even though that choice was among the most valuable decisions in the episode. This means a benchmark that only reports final success can hide the fact that the agent's skill-selection policy is essentially untrained—it might succeed by luck or by reading many skills, not by genuinely learning to pick the right one.
Why doesn't just rewarding the final outcome teach skill selection?
The default remedy—outcome-rewarded reinforcement learning over a candidate slate—cannot teach skill selection, for a structural reason. In a broadcast, sequence-level advantage, the credit for a successful episode is spread across all tokens, so the skill-naming tokens get a tiny fraction of the update [1]. The paper's audits of a completed run's training artifacts confirm three properties: the credit share is vanishing, the sign becomes increasingly wrong as trajectories lengthen, and all three worsen monotonically with horizon [1]. This means that in long-horizon tasks, the skill-selection signal is not just weak—it actively misleads the policy.
The practical consequence is that an agent trained with outcome reward alone may appear to improve on a benchmark, but its skill-selection behavior is not actually being shaped. The 2026 study shows that when the same budget is spent on a dedicated action-local advantage for the skill-naming tokens (SkillGate), a 9B-parameter policy jumps from 40.8% to 53.2% trial success across five agentic benchmarks with a 16-candidate skill slate [1]. That 12.4-point gain—roughly a 30% relative improvement—comes from fixing the credit assignment for the selection decision, not from better execution. The same method also cut exposure to misleading candidates by two-thirds and read fewer skills, indicating that the agent learned to be more selective, not just more successful.
Is skill selection a separate problem from skill execution?
Yes—and the evidence from other work points the same way. A 2023 robotics study (LEAGUE) combines task planning with RL-based skill learning, and explicitly separates the symbolic interface (which skill to use) from the low-level skill execution [2]. It shows that learning skills in-situ of a task planner—where the planner guides which skill to learn and when—allows the system to grow its capability and reuse skills across new tasks [2]. This supports the idea that skill selection is a distinct decision layer that benefits from its own training signal, not something that emerges automatically from execution-level rewards.
A 2021 study on skill-sequence-dependent policies makes a complementary point: long-horizon tasks are not Markovian in the current observation alone—the sequence of skills matters [3]. Their hierarchical policy uses a high-level skill policy that explicitly conditions on skill sequences, and a low-level parameter policy that responds to observations, and it learns significantly faster than standard PPO (Proximal Policy Optimization) [3]. Together, these studies converge on the same conclusion from different angles: skill selection is a learnable, sequence-aware decision that requires dedicated structure, and benchmarks that only measure final success can mask failures in this layer. The 2026 SkillGate paper is the most direct evidence, because it isolates the selection signal and shows a large performance jump from training it explicitly [1].
About These Sources
This answer is built on 3 studies (2 peer-reviewed, 1 preprint) — published from 2021 to 2026, 1 from 2024 or later, 1 in Q1–Q2 journals — selected as the most relevant from 3 studies that passed quality screening, drawn from 29 papers retrieved from a database of over 500 million.
Sources used in this answer
SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents
Identifies 'selector credit starvation' in outcome-rewarded RL for skill selection, showing it worsens with horizon; proposes SkillGate, which lifts a 9B policy from 40.8% to 53.2% trial success on five benchmarks with a 16-candidate slate, while cutting exposure to misleading candidates by two-thirds.
LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation
LEAGUE integrates task planning with RL-based skill learning, using a symbolic interface to guide which skills to learn and when, enabling skill reuse and continuous capability growth across four simulated task domains and transfer to a physical robot.
Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks
Proposes a skill-sequence-dependent hierarchical policy for long-horizon manipulation, where a high-level skill policy conditions on skill sequences and a low-level parameter policy responds to observations, learning significantly faster than PPO and task schema methods in simulation.