How well does ES assign credit across long horizons?
The first thing a fair evaluation must measure is credit assignment—how the training method figures out which actions led to a reward, especially when rewards are sparse and delayed. Evolution strategies (ES) have a structural advantage here: they perform trajectory-level parameter attribution without decomposing rewards across horizons, which scales better than RL as the horizon grows [3]. A 2025 study demonstrated this by showing ES outperformed established RL implementations on long-horizon and delayed-reward tasks, while also being more robust across different base models and less susceptible to reward hacking [5]. This means a fair evaluation should include tasks where rewards only appear after many steps, and should compare ES against RL on those tasks, not just on short-horizon ones.
However, ES is not a silver bullet. A 2026 paper on memory-augmented agents points out that when agents write, update, or delete memories across sessions, trajectory-level comparisons become unfair because rollouts no longer share the same intermediate memory state [1]. This is a critical caveat: if your evaluation uses group-relative methods like GRPO (group-relative policy optimization), it may give biased credit signals for long-horizon memory operations. A fair evaluation must therefore include scenarios where memory changes over time, and must check whether the training method can handle that without bias. The paper proposes a solution—combining local and global group-relative optimization—but the key point for evaluation is that you need to test memory-heavy tasks separately from simple long-horizon ones.
Does the trained agent generalize beyond its training environment?
A fair evaluation must ask whether the agent has learned transferable skills or just memorized environment-specific tricks. A 2026 study trained a model on 363 long-horizon tasks across 27 categories and then tested it on five external benchmarks, finding improvements of +2.8 to +9.6 percentage points over the base model—even on software-engineering benchmarks that were not in the training data [7]. This is strong evidence that ES-trained agents can generalize, but it also highlights the need to evaluate on held-out benchmarks that share no task templates or graders with training. The study's paired-trajectory analysis identified four recurring behavioral differences—like more careful goal formation and verifying completion—that transferred across domains, suggesting that evaluation should look at behavior, not just final scores.
Robustness is another dimension that a fair evaluation must include. Long-horizon agents are vulnerable to attacks that exploit multi-turn interactions, such as memory poisoning or objective drifting, and a 2026 benchmark found that even state-of-the-art agents remain highly susceptible, with single-turn defenses failing to mitigate these threats [6]. This means an evaluation should include adversarial scenarios to see if ES training makes agents more or less robust. Additionally, a 2024 study on very long-term conversational memory found that LLMs struggle with long-range temporal and causal dynamics, and even with long-context or retrieval-augmented generation, they lag behind human performance [4]. So a fair evaluation should test memory consistency and temporal reasoning, not just task completion.
Is ES training practical for large models and long horizons?
A fair evaluation must also measure the practical cost of training. ES offers a major advantage in scalability: it enables full-parameter optimization with only inference-level GPU memory, making it possible to fine-tune large models that would be impractical with RL's backpropagation-based stack [3]. A 2025 study confirmed this by demonstrating ES fine-tuning at the billion-parameter scale without dimensionality reduction [5]. This means an evaluation should report GPU memory usage and training time, not just final performance, because a method that achieves slightly lower scores but at a fraction of the cost might be more practical.
However, ES has a known weakness: high sample complexity, especially over long horizons. A 2022 study addressed this by using auxiliary short-episode tasks to transfer skills to the full-length task, reducing agent-environment interaction data requirements [2]. This suggests that a fair evaluation should measure sample efficiency—how many environment interactions are needed to reach a given performance level—and should consider whether the method can leverage auxiliary tasks to speed up learning. The study's adaptive resource allocation strategy also implies that evaluation should track how computational resources are used across auxiliary and main tasks, as that affects overall efficiency.
About These Sources
This answer is built on 7 studies (2 peer-reviewed, 5 preprints) — published from 2022 to 2026, 6 from 2024 or later, 1 in Q1 journals — selected as the most relevant from 13 studies that passed quality screening, drawn from 39 papers retrieved from a database of over 500 million.
Sources used in this answer
Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents
Memory-R2 introduces LoGo-GRPO to address unfair credit assignment in memory-augmented LLM agents, combining local and global group-relative optimization to improve supervision for memory construction.
Multitask Neuroevolution for Reinforcement Learning With Long and Short Episodes
NuEMT transfers skills from short-episode auxiliary tasks to long-horizon RL tasks, reducing sample complexity and achieving data-efficient evolutionary RL on continuous control tasks.
Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements
Agentic ESOpt demonstrates that ES enables full-parameter fine-tuning of large LLMs with minimal GPU memory, improving WebArena-Lite performance by 6.69% over a No Skill baseline.
Evaluating Very Long-Term Conversational Memory of LLM Agents
LoCoMo, a dataset of very long-term conversations (up to 32 sessions, 600 turns), shows that LLMs struggle with long-range temporal and causal dynamics, and even with long-context or RAG, they lag behind human performance.
Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
ES fine-tuning at billion-parameter scale outperforms RL on long-horizon and delayed-reward tasks, with improved robustness and reduced reward hacking, across diverse base LLMs.
AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks
AgentLAB, a benchmark with 644 security test cases across 28 environments, shows that LLM agents remain highly susceptible to long-horizon attacks, and single-turn defenses fail to mitigate them.
Cross-Benchmark Generalization in Long-Horizon Agents
Post-training on 363 long-horizon MCP tasks improved performance on five external benchmarks by +2.8 to +9.6 percentage points, including software-engineering tasks not in the training data.