Why do agents repeat mistakes in long tasks? The core problem is credit assignment, not memory.
The biggest reason long-horizon agents repeat mistakes is that the standard training signal—rewarding the whole episode—can't tell the agent which decision caused the failure. In a long task, the few tokens that name the chosen skill carry a vanishingly small share of the loss, and the credit they inherit is increasingly wrong-signed as the trajectory lengthens. This is called "selector credit starvation," and it's structural: a correct skill choice gets punished if the execution after it fails, even though the choice itself was the most valuable decision [2]. This is why simply adding more memory or more training data doesn't fix the problem—the agent never learns which skill to pick.
The fix is to give the skill-selection decision its own reward signal, separate from the execution. SkillGate does this by partitioning the token support into two disjoint credit channels: outcome credit reaches only execution tokens, and a separate action-local advantage reaches exactly the skill-naming tokens, positive only when the trajectory's single read is the correct one [2]. On five agentic benchmarks with a 16-candidate skill slate, this lifted a 9B-parameter policy from 40.8% to 53.2% trial success—a 12.4-point jump—while cutting exposure to misleading candidates by two thirds and reading fewer skills [2]. The same budget spent on outcome reward alone barely moved the needle, confirming that the problem is the training signal, not the model size.
Even with good training, memory can become polluted—and that's when agents repeat mistakes.
Long-horizon agents often use external memory to remember what worked and what didn't. But if that memory fills with noisy or failed attempts, the agent can retrieve the wrong example and repeat the same mistake. In a controlled stress test with 75% synthetic distractors, unbounded memory and a simple FIFO (first-in, first-out) eviction policy both degraded badly: Precision@5 (the fraction of relevant memories in the top 5 retrievals) dropped from 20.2% to 12.4% for unbounded memory and from 15.8% to 3.8% for FIFO [1]. In plain terms, the agent was pulling up the wrong memories most of the time.
A selective retention policy—TraceRetain—scores each memory entry by interpretable features like success, age, access frequency, and redundancy, and evicts the lowest-scoring ones at capacity. Under the same noise, TraceRetain kept Precision@5 essentially unchanged (16.9% to 16.6%) and preserved 97/100 task success [1]. The mechanism is telling: unbounded memory had the highest mean similarity (0.87) but the lowest precision, meaning failed distractors were embedding close to the query—so the agent was confidently retrieving the wrong thing. This shows that memory hygiene is as important as the training signal for avoiding repeated mistakes.
When does in-policy skill selection actually help? Only when the task is long enough and noisy enough.
The benefits of these methods are not universal. On clean, saturated benchmarks, memory-augmented policies with different retention policies all perform within statistical error of each other—the differences fall within Wilson 95% confidence intervals [1]. In other words, if the task is easy and the memory stream is clean, any reasonable memory policy works, and the extra complexity of selective retention buys little. The advantage only appears when the stream contains noise or distractors, which is exactly the situation where mistakes can compound over long horizons.
Similarly, the credit-assignment fix matters most when the task is long. The paper on SkillGate shows that the credit starvation problem worsens monotonically with horizon [2]. So for short tasks, outcome-rewarded RL might be sufficient, but for long multi-step tasks, the skill-selection decision becomes too important to leave to a vanishing share of the loss. This aligns with findings from robotics: methods that explicitly chain skills and learn transition policies (like curriculum-guided skill chaining [3] or skill-sequence-dependent policies [5]) outperform flat RL approaches on long-horizon tasks, because they break the problem into manageable pieces and learn when to switch. The takeaway: in-policy skill selection is not a silver bullet, but it is necessary when tasks are long and the environment is noisy.
About These Sources
This answer is built on 5 studies (3 peer-reviewed, 2 preprints) — published from 2021 to 2026, 3 from 2024 or later, 1 in Q1 journals — selected as the most relevant from 6 studies that passed quality screening, drawn from 41 papers retrieved from a database of over 500 million.
Sources used in this answer
Selective Memory Retention for Long-Horizon LLM Agents
In controlled noisy-write stress tests (75% synthetic distractors), unbounded memory and FIFO eviction degraded Precision@5 from 20.2% to 12.4% and 15.8% to 3.8%, respectively, while TraceRetain kept it nearly unchanged (16.9% to 16.6%) and preserved 97/100 task success; on clean benchmarks, retention policies did not differ significantly.
SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents
Identified 'selector credit starvation' as a structural failure of outcome-rewarded RL for skill selection, and proposed SkillGate, which partitions credit into separate channels; on five agentic benchmarks with a 16-candidate slate, SkillGate lifted a 9B policy from 40.8% to 53.2% trial success, while cutting exposure to misleading candidates by two thirds.
Curriculum-guided skill learning for long-horizon robot manipulation tasks
Proposed a curriculum-guided skill-chaining algorithm that learns transition policies between skills from a single demonstration, achieving similar performance to data-hungry methods on a robotic manipulation task and enabling zero-shot skill reuse.
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
Introduced EMBER, a model-based RL method that learns a repertoire of primitive skills with a success classifier for retrying, enabling a Franka robot to complete three long-horizon visuomotor tasks at 85% success rate, sequencing up to 12 skills with 14 unique primitives.
Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks
Proposed a skill-sequence-dependent hierarchical policy that uses both observation and skill sequences, solving a long-horizon task significantly faster than PPO and task schema methods in simulation.