Actor-Curator: Optimizing Reinforcement Learning Post-Training via Policy-Improvement Bandits
Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
Actor-Curator is a fully automated curriculum learning framework for Reinforcement Learning (RL) post-training of Large Language Models. It introduces a neural curator that dynamically selects training problems by optimizing for expected policy performance improvement, achieving up to 80% training speedup and significant accuracy gains (e.g., +30.5% on ARC-1D).
Executive Summary
Reinforcement Learning (RL) has become the gold standard for post-training Large Language Models (LLMs) in reasoning and alignment tasks. However, the sample efficiency of RL is notoriously low when training on massive, heterogeneous datasets. The choice and order of training problems—the curriculum—can make or break the stability and speed of convergence.
Actor-Curator presents a paradigm shift: it automates the curriculum by training a "neural curator" alongside the LLM (the "actor"). Instead of relying on human-labeled difficulty or success rates, it optimizes a bandit-style objective grounded in Policy Improvement Theory. This work isn't just a heuristic tweak; it provides a principled, scalable framework that achieves massive efficiency gains, such as an 80% training speedup on symbolic logic tasks.
The Problem: The Inefficiency of Uniformity
In a typical RL rollout, we sample problems uniformly. But as the policy improves, many problems become redundant:
- Too Easy: The agent already knows the answer; the gradient is near zero.
- Too Hard: The agent never finds a reward signal; no learning occurs.
- Static Heuristics: Methods like "prioritize 50% success rate" are better than uniform sampling but don't account for how a specific problem affects the overall distribution of the model's capabilities.
Methodology: Bridging Bandits and Policy Improvement
The genius of Actor-Curator lies in its objective function. The team derived a per-problem utility based on the Performance Difference Identity.
1. The Core Objective
The curator's goal is to maximize the cumulative performance gains: By using importance sampling, they decompose this into a per-problem utility that measures how much training on problem will shift the model toward higher rewards globally.
2. Scalable Online Stochastic Mirror Descent (OSMD)
To manage the massive problem bank, they use a two-stage sampling loop:
- Proposal: Sample a large "candidate batch" from the dataset.
- Curation: A lightweight neural network (e.g., Qwen-0.6B) reweights these candidates to select the final training batch.
Figure 1: The co-adaptive online training loop where the curator adapts to the actor's evolving capability.
Experiments and SOTA Results
The authors tested Actor-Curator against uniform sampling and strong baselines like PCL (Prompt Curriculum Learning) and SEC (Self-Evolving Curriculum) across mathematical and symbolic reasoning benchmarks.
Performance Gains
Actor-Curator showed its greatest strength in "Hard" subsets:
- ARC-hard: +58.9% improvement over the strongest baseline.
- AIME24: +28.6% improvement.
- Countdown-hard: +12.6% improvement.
Efficiency and Signal Quality
Why does it work? Analysis shows that curated problems induce larger actor gradient norms, meaning the curator is successfully finding problems that force the model to learn more per step.
Figure 2: Performance on Countdown benchmarks showing faster convergence and higher ceilings compared to uniform sampling.
Critical Insight: The "Why"
Unlike previous methods that regress success probabilities, Actor-Curator's use of Mirror Descent allows it to maintain a stable distribution while exploring the "utility frontier." As shown in the "Difficulty Progression" analysis, the model naturally learns to start with easy tasks and "curate" a path toward harder problems without any human guidance.
Figure 3: The automated difficulty progression. The curator autonomously shifts focus as the actor matures.
Limitations & Future Outlook
While the overhead is small (~9% extra wall time), it does require a second model (the curator). Furthermore, the effectiveness depends on a verifiable reward signal, making it currently ideal for math, code, and logic puzzles.
The future of RL post-training likely involves these "agentic" data curation layers, where the dataset is no longer a static folder of files, but a dynamic, optimized stream calibrated to the model's current "zone of proximal development."
takeaway
Actor-Curator proves that what you learn is just as important as how you learn. By treating data curation as a first-class optimization problem, we can unlock the potential of LLMs much faster than through brute-force computation.