Agentic ESOpt: Scaling Full-Parameter Tuning of Long-Horizon LLM Agents with Inference-Level Memory
Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements
This paper introduces Agentic ESOpt, a full-parameter Evolution Strategies (ES) fine-tuning and test-time optimization framework designed specifically for long-horizon Large Language Model (LLM) agents. By substituting heavyweight backpropagation with forward-only parameter perturbations, Agentic ESOpt enables full-parameter fine-tuning of multi-billion parameter models with minimal, inference-level GPU memory while overcoming the severe credit assignment degradation suffered by policy gradient methods as the interaction horizon grows. On challenging multi-turn benchmarks, Agentic ESOpt outperforms Agentic GRPO by 12.50% on 15-horizon Sudoku, enables full-parameter tuning of Qwen3.5-27B on WebArena-Lite (+6.69%), and improves test-time search across 28 of 36 algorithmic heuristic design settings.
Executive Summary
TL;DR
Agentic ESOpt re-evaluates the optimization landscape of multi-turn Large Language Model (LLM) agents by substituting backpropagation-based Reinforcement Learning (RL) with parameter-space Evolution Strategies (ES). By perturbing model weights directly and updating them via normalized trajectory returns, Agentic ESOpt achieves full-parameter fine-tuning using only inference-level GPU memory (e.g., 8.41 GB for a 4B model vs. 58.88 GB for GRPO). Crucially, because ES evaluates trajectory-level parameter perturbations rather than accumulating per-step action gradients, its variance does not scale linearly with the interaction horizon , outperforming state-of-the-art RL baselines (PPO and GRPO) on long-horizon reasoning and enabling full-parameter adaptation of 27B-scale web agents.
Academic Coordinate & Positioning
In the spectrum of LLM post-training:
- Single-Turn RL (PPO, GRPO, R1-Zero): Dominates single-turn mathematical and code reasoning where token-level rewards or verifiers are immediate.
- Agentic / Multi-Turn RL (Turn-PPO, Group-in-Group Policy Optimization): Struggles with activation memory explosion, critic warm-up, and long-horizon credit assignment degradation under sparse feedback.
- Prompt/Skill Evolution (Trace2Skill, SkillOpt, Reflexion): Lightweight and gradient-free, but inherently constrained to behaviors accessible within the frozen model's parameter manifold.
- Agentic ESOpt: Opens a new paradigm by establishing ES not just as a cheaper alternative to RL, but as a structurally superior post-training and test-time co-evolution engine for long-horizon, multi-turn agentic environments.
Problem & Motivation: The Breakdown of Agentic RL
Fine-tuning LLMs for multi-turn agentic tasks (e.g., multi-step tool invocation, interactive web navigation, and iterative algorithmic design) exposes two fundamental failure modes of traditional policy-gradient RL:
1. Prohibitive Memory Footprint for Full-Parameter Tuning
In multi-turn agentic RL, rollouts involve extended conversation contexts, tool outputs, and branching trajectories across hundreds of tokens per turn. Gradient-based methods (PPO/GRPO) must cache forward activations across these entire trajectories, maintain reference model copies, and compute backpropagation graphs across the full model parameter space. As a result, fine-tuning a moderate 4B model under GRPO demands ~58.88 GB of VRAM, while PPO requires ~89.40 GB. For 27B+ parameter backbones, full-parameter multi-turn RL becomes virtually intractable on standard academic/industrial compute nodes (e.g., 4 H100 80GB GPUs).
2. Horizon-Dependent Variance Explosion ()
Consider a multi-turn trajectory of horizon receiving a sparse scalar terminal return . A standard policy gradient estimator with baseline evaluates:
abla_ heta \log \pi_ heta(a_t \mid s_t)$$ Under standard weak-correlation assumptions where intermediate actions contribute noisy signals toward the final sparse reward, the policy score terms accumulate, yielding an estimator covariance that scales linearly with the interaction horizon: $$\mathrm{Var}[\widehat{g}_{\mathrm{PG}}] \approx \mathrm{Var}[R(\mathbf{a})] \cdot \mathrm{Var}\left[\sum_{t=1}^{H} abla_ heta \log \pi_ heta(a_t \mid s_t)\right] \propto H$$ As $H$ grows, token-level policy gradients become severely misaligned with true trajectory-level credit, leading to policy collapse, excessive trajectory lengths, or getting trapped in suboptimal local attractors. --- ## Methodology: Agentic ESOpt Architecture & Mechanics  *Figure 1: High flexibility of Agentic ESOpt. Parameter perturbations are evaluated via forward-only rollouts with scalar feedback, simultaneously enabling prompt-parameter co-evolution and skill synthesis via shared traces.* ### 1. Parameter-Space Objective & Gradient Derivation Instead of differentiating through intermediate environment transitions and action probabilities, Agentic ESOpt defines the expected trajectory reward over a Gaussian-smoothed parameter objective: $$J_\sigma( heta ; c) = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)} \left[ J( heta + \sigma \epsilon ; c) \right]$$ Applying the log-derivative trick directly in parameter space yields the ES pseudo-gradient:abla_ heta J_\sigma( heta ; c) = \frac{1}{\sigma} \mathbb{E}{\epsilon \sim \mathcal{N}(0, I), au \sim \pi{ heta + \sigma \epsilon}(\cdot \mid c)} \left[ R( au) \epsilon \right]$$
Because one perturbation governs the entire multi-turn episode, the estimator:
does not sum over action scores. Parameter attribution assigns the terminal return directly to a coherent policy shift, decoupling estimator variance from interaction horizon length.
2. In-Place Seed Replay & Reward Normalization
To eliminate the memory overhead of storing model weights or multi-gigabyte noise tensors:
- Agentic ESOpt samples pseudo-random noise seeds.
- Perturbations are applied in-place during the forward rollout.
- The perturbation is reverted immediately in-place after trajectory generation.
- Trajectory returns are standardized via population -score:
- Parameters are updated using the reconstructed noise seeds:
This forward-only loop requires strictly inference-level GPU memory (the memory necessary to load the model weights and run forward generation).
3. Cosine Decay Schedule on Perturbation Scale
A non-zero induces a Gaussian smoothing bias characterized by the Taylor expansion:
abla_ heta^2 J( heta ; c)\right) + \mathcal{O}(\sigma^4)$$ The Laplacian term $\mathrm{Tr}( abla_ heta^2 J)$ acts as an implicit regularizer favoring flatter parameter regions. To exploit broad exploration early while ensuring fine-grained convergence and minimizing objective bias later, Agentic ESOpt introduces a cosine decay schedule: $$\sigma_t = \sigma_T + (\sigma_0 - \sigma_T) \frac{1 + \cos(\pi t / T)}{2}$$ * **Train-time adaptation**: Retains a non-zero $\sigma_T$ (e.g., $5 imes 10^{-4}$) to prevent overfitting and retain neighborhood flatness. * **Test-time search**: Decays $\sigma_T o 0$ to eliminate objective bias when optimizing for the best candidate on a specific instance. --- ## Controlled Scalability: The Sudoku Benchmark To isolate the relationship between horizon length $H$ and optimization efficacy, the authors designed a controlled multi-turn Sudoku benchmark where the minimum successful horizon $H^* \in \{5, 10, 15\}$ corresponds to the number of masked cells.  *Table 1: Sudoku final success rates (%) and GPU memory requirements across minimum horizons $H^*$.* ### Key Empirical Findings 1. **Horizon Crossover**: At short horizons ($H^* = 5$), RL and ES are competitive (85.42% for GRPO vs. 89.58% for ESOpt). However, at $H^* = 15$, GRPO performance drops to 40.63% and PPO collapses to 0.00%, while Agentic ESOpt maintains **53.13%** (+12.50% over GRPO). 2. **Trajectory Bloat in RL**: Under sparse rewards, GRPO agents frequently exhaust the maximum turn budget (45 turns) without resolving the board, whereas Agentic ESOpt agents consistently converge near the minimal required turns (15.41 turns). 3. **Hardware Efficiency**: Agentic ESOpt consumes only **8.41 GB** of VRAM compared to **58.88 GB** for GRPO and **89.40 GB** for PPO. While ES uses $G=32$ rollouts per update (vs. 8 for GRPO), the elimination of backward passes and reference model evaluations means model-side FLOPs remain equivalent ($\approx 1 imes$), and end-to-end wall-clock training time is substantially faster (9.4h vs. 19.0h at $H^*=15$). --- ## Empirical Evaluation Across Agentic Domains ### 1. ReAct-Style Tool Use: Math Reasoning & DocVQA Evaluating Qwen3.5-4B on complex multi-turn tool interaction (Python execution for DAPO/AIME 2026 and image crop/OCR tools for DocVQA):  *Table 2: Performance on ReAct Math and DocVQA under No Skill and Trace2Skill conditions.* * **Substantial Gains Over GRPO**: On AIME 2026, Agentic ESOpt achieves **70.8% Mean@4** (vs. 58.3% for GRPO and 55.8% base). * **Composition with Skill Space**: When combined with Trace2Skill (distilling procedural skills from failed/successful traces), Agentic ESOpt reaches **77.3%** on DAPO and **71.7%** on AIME 2026, establishing seamless synergy between parameter updates and external prompt optimization.  *Figure 2: Best-of-$k$ Pass@$k$ curves across sampling budgets $k \in \{1, \dots, 32\}$. Agentic ESOpt consistently preserves higher distribution coverage than GRPO.* --- ### 2. Large-Model Scalability: Qwen3.5-27B on WebArena-Lite Full-parameter fine-tuning of a 27B model on interactive web environments (GitLab, CMS, Reddit, Maps, OSS) is practically inaccessible for multi-turn RL on a 4$ imes$ H100 80GB cluster. Agentic ESOpt's forward-only memory footprint made full-parameter training trivial ($G=8$).  *Table 3: WebArena-Lite goal-conditioned web navigation success rates (%).* * **Direct Policy Improvement**: Agentic ESOpt boosts the unaugmented Qwen3.5-27B base model from **29.47% to 36.16%** (+6.69%), outperforming closed-source GPT-5.4 (34.14%). * **Post-Hoc Skill Distillation**: Combining ESOpt with Trace2Skill elevates the benchmark score to **36.36%**, demonstrating that parameter updates improve the quality of downstream distilled skills. --- ### 3. Test-Time Compute: Automatic Heuristic Design (AHD) In Automatic Heuristic Design (e.g., Evolution of Heuristics / EoH for NP-hard problems like TSP, Knapsack, and Bin Packing), the LLM generates heuristic code evaluated inside a black-box solver. Traditional test-time compute freezes the model and only searches over code candidates. Agentic ESOpt inserts online parameter updates directly into the mutation operators ($m_1, m_2$) of EoH: * Across 12 constructive and ACO-style combinatorial tasks, Agentic ESOpt + EoH improves upon matched baselines in **28 out of 36 settings** under identical evaluation budgets. * Parameter updates execute on-the-fly, adding only **9.7%–18.0%** runtime overhead without requiring a separate offline post-training pipeline. --- ## Critical Analysis & Insights ### Population Scaling Laws with Backbone Capability A critical preliminary finding of this work is the relationship between model scale and ES population size $G$: ``` Table: Population Sensitivity on 15-turn Sudoku (Success Rate %) ---------------------------------------------------------------- Backbone G=8 G=16 Δ (Relative Improvement) ---------------------------------------------------------------- Qwen3.5-4B 2.95% 22.92% +677.0% Qwen3.5-9B 30.21% 30.21% 0.0% ---------------------------------------------------------------- ``` * **Theoretical Implication**: For smaller models (4B), the local parameter landscape contains high variance; thus, a large population ($G \ge 16$) is necessary to average out gradient noise. * **Density of Competence**: Stronger pre-trained backbones (9B+) possess denser manifolds of capable behaviors. A significantly smaller population ($G=8$) is sufficient to obtain high-signal descent directions, opening a viable path for applying ES to frontier models with very few sampled perturbations. ### Limitations 1. **Environment Evaluation Bottlenecks**: Agentic ESOpt trades backpropagation FLOPs for more forward trajectory rollouts. In domains where environment execution is prohibitively slow or costly (e.g., physical robot simulators or heavy web sandboxes), the rollout cost can dominate model compute savings. 2. **Dense vs. Sparse Parameter Updates**: While ES updates modify all parameters, empirical measurements confirm that **96.26%** of parameter updates remain bounded within $|\Delta heta| \le 1.5 imes 10^{-3}$, mitigating catastrophic forgetting concerns during targeted adaptation. --- ## Conclusion & Outlook Agentic ESOpt fundamentally shifts the perspective on Evolution Strategies in modern AI post-training: * **Memory Scalability**: Forward-only updates enable full-parameter tuning of 27B+ LLMs at inference VRAM limits. * **Credit Assignment**: Decoupling variance from trajectory horizon length makes ES structurally superior to RL under sparse, long-horizon feedback. * **Universal Co-Evolution**: The black-box interface allows parameters, prompts, and skills to be co-optimized within unified train-time and test-time loops. Future work exploring quantization-aware ES perturbations, adaptive noise covariance matrices, and coupled multi-step skill-parameter loops will likely establish ES as a standard component of large-scale agentic post-training frameworks.