Can scaling reinforcement learning compute drive foundation models toward self-improvement?

Three levers of RL compute, not one

The paper treats "scaling RL compute" as three separate levers rather than a single number. The first is training scale: fully asynchronous training consumes 1,568 samples and 2.7–3.7B tokens per step, at context lengths up to 1M. The second is environment scale: RL tasks span code, general, visual and cybersecurity domains, run under a mixture of agent harnesses — the scaffolding that drives a model through multi-turn tool use. The third is grader scale: instead of a binary pass/fail on test cases, groupwise agentic grading compares solutions inside each group and converts those comparisons into finer reward signals through Groupwise Reward Synthesis and Groupwise Advantage Redistribution.

This decomposition matters for reading the results. A benchmark improvement in this paper cannot be attributed to "more RL" in the abstract; it is the joint product of larger batches, more varied task-and-harness combinations, and denser reward information. The authors also state that varying both harness and task improves generalization, which is why harness diversity is treated as a scaling axis rather than an implementation detail.

Two model sizes carry the experiments: MiMo-V2.6-Pro, a 1.02T-parameter Mixture-of-Experts model with 42B active parameters, and MiMo-V2.6-Flash, a 310B-parameter Mixture-of-Experts model with 15B active parameters. Both sit on a hybrid backbone that interleaves local sliding-window attention with global attention, and both pass through an agent-centric mid-training phase before RL, intended to widen the exploration space available during post-training.

Gains appear across four task domains

The clearest evidence for the paper's claim is the training-dynamics figure: as RL training steps increase, both Pro and Flash improve on DeepSWE v1.1, SWE-Bench Pro and MiMo Code Bench for coding, AutomationBench v1.0.6 for general workflows, MiMo Visual Coding for visual design tasks, and MiMo Cyber Bench for cybersecurity. The authors describe these gains as holding consistently across domains rather than concentrating in one.

That consistency is the paper's main argument that scaled RL produces general agentic capability rather than a narrow specialization. It is also a claim about training curves, not about final superiority: the figure shows scores rising with steps for the two MiMo models, and the separate benchmark table places them against prior-generation and frontier systems.

The final benchmark table reports MiMo-V2.6-Pro and Flash alongside MiMo-V2.5-Pro, Claude Opus 5, GPT-5.6 Sol and Claude Fable 5. The authors summarize the outcome as performance comparable to frontier models across a broad range of agentic tasks, and as substantial improvement over MiMo-V2.5. Individual rows vary: for example, Pro scores 71.9 on DeepSWE v1.1 against 74.0 for Claude Opus 5, while Flash scores 95.1 on CyberGym against 40.0 for MiMo-V2.5-Pro. Readers comparing systems should use the per-benchmark rows rather than the summary sentence.

Groupwise grading rewards quality, not just passing

Outcome rewards give the same credit to every solution that passes a test, which is coarse supervision for long-horizon agent tasks. The paper's response is groupwise agentic grading: solutions within a group are compared against each other, so differences in problem-solving quality and behavior become visible, and GRS and GAR turn those distinctions into learning signals. The stated effect is twofold — more accurate reward signals for long-horizon tasks, and steering the model toward shorter, more token-efficient solutions.

This is a mechanism claim about the reward channel, and it is worth separating from the benchmark numbers. The paper does not report a controlled ablation isolating grader compute alone in the evidence available here; the token-efficiency and accuracy effects are presented as consequences of the grading design within the overall recipe. What the paper does report concretely is that the reward signal is no longer binary, and that the model is pushed toward shorter solutions rather than any passing solution.

A second safeguard sits alongside grading: a multi-layer defense against reward hacking, plus freezing the MoE router to keep expert loads stable during large-scale training. The router freeze is a stability measure, not a capability measure, and the paper presents it that way.

Infrastructure turns long rollouts into trainable batches

Scaling batch size on agentic tasks is an engineering problem before it is a research one. The paper describes a unified trajectory representation organized as Sample, Sequence, Context and Segment, where only model-generated turns contribute to the loss. A Penalty Module applies loss masking and advantage shaping so that infrastructure failures are kept out of the training signal rather than taught to the model as behavior.

Around that data model sit several systems: a Harness Pool hosting multiple agent harnesses at high concurrency, a Payload Porter that buffers and transfers large trajectories carrying routing and multimodal payloads, a Sample Mixer that stabilizes per-task sample composition in training batches, and alignment of MoE routing and top-p sampling candidate sets between training and inference engines. The paper also adopts Group Relative Policy Optimization with prompt-mean loss aggregation, which it says prevents response length from growing too quickly during RL, and decoupled clipping bounds tuned at runtime based on policy entropy.

The failure analysis is unusually concrete and worth reading as a scope condition. Reported interruptions include GPU-memory double-bit errors, a Kubernetes failure that crashed Cyber-task pods between steps 15 and 16 for Flash, an unreachable grader after step 14 for Pro, rollout-length bias in partial-rollout dispatch that exhausted GPU and pinned host-memory KV pools, MoE imbalance where one expert-parallel rank received over 30× the mean token load within a micro-batch, and CPU out-of-memory during packing late in Flash training. These are the practical costs of the scale the paper claims.

A 9B distilled model reproduces the RL gains

The paper's most checkable transfer test uses MiMo-V2.6-Distill-Qwen-9B, a small model distilled from MiMo-V2.6 and then trained with GRPO on domain-specific checkpoints. Table 6 reports Qwen3.5-9B, the distilled model after SFT, and the same model after RL: SWE-bench Verified moves 60.0 → 61.1 → 66.2 (avg@3), SWE-bench Pro 32.0 → 44.6 → 47.6, MiMo Code Bench (mini) 19.5 → 51.6 → 59.9, MiMo Cyber Bench (mini) 5.7 → 31.3 → 47.0, and MiMo Visual Coding (mini) 61.7 → 64.0 → 72.4 (avg@1).

The multi-harness experiment addresses a different question: whether training across harnesses helps on harnesses the model never trained on. Table 7 reports mean scores over seven harnesses, four used in training and three held out. On SWE-bench Verified the mean rises from 62.3 for the distilled model to 65.7 after multi-harness RL; on SWE-bench Pro from 44.4 to 46.5; on MiMo Code Bench (mini) from 53.1 to 59.0. Held-out harness columns improve too, which is the paper's evidence that harness variation contributes to generalization rather than to harness-specific overfitting.

These are the numbers to weigh if the question is whether the recipe transfers beyond the two flagship models. They are also bounded: the distilled model is a 9B Qwen-based student, the code results in Table 6 use single-harness RL, and the scores are benchmark-specific averages with the stated avg@3 or avg@1 aggregation.

Self-improvement remains the framing, not the finding

The paper opens with recursive self-improvement as motivation and closes by calling its work a practical step toward model self-improvement. What the evidence supports is narrower: scaling RL compute along batch size, environment diversity and grader compute, with the described infrastructure and stability measures, produced rising benchmark scores during training and competitive final scores against frontier models on the reported agentic benchmarks. No experiment in the evidence isolates a loop in which the model improves itself without external task environments, verifiers or graders.

The recipe also depends on components the paper had to build: agent-centric mid-training to widen exploration, a hybrid attention backbone, frozen MoE routing, reward-hacking defenses, and a grading system that requires comparing solutions within groups. Where reliable verification is hard to design — the paper names long-horizon game development, scientific research and embodied intelligence — it does not rely on RL rewards but on Multi-Prefix Multi-Teacher On-Policy Distillation, combining teachers trained with mixed RL on verifiable tasks and SFT teachers on synthetic demonstrations for open-domain tasks. That is an admission that the RL path is bounded by verifiability.

The open-source release — MiMo-V2.6-Distill-Qwen-9B, curated task environments with verifiers, an end-to-end RL framework and composable mini-harnesses — is the part of the paper most directly usable by other researchers, because it lets them test whether the same three-lever scaling recipe reproduces on their own tasks.

Judge the recipe by its three levers together

The paper's practical priority is that batch scale, environment and harness diversity, and grader compute were scaled as a package, and the reported gains belong to that package. A reader evaluating whether to adopt the approach should ask which of the three levers their own setup can actually move: whether they can sustain thousands of long-horizon rollouts per step, whether they can vary harnesses as well as tasks, and whether they can afford groupwise grading instead of binary test outcomes.

The most decision-relevant comparison in the paper is the multi-harness result on held-out harnesses, because it separates generalization from fitting the training configuration. The second is the distilled 9B model's progression from SFT to RL, because it shows the gains are not exclusive to trillion-parameter models. The self-improvement framing should be read as the authors' stated direction; the measured contribution is a scaled, reproducible RL recipe whose benefits appear across domains and harnesses, with the token-efficiency effect attributed to the grading design rather than isolated in a controlled comparison.

Open questions the paper leaves for reproduction

Several things a reader might want are not settled by the evidence here. The paper reports that groupwise grading steers the model toward shorter, more token-efficient solutions but does not, in the available text, give a separate measurement of token counts per solution with and without grading. The benchmark table mixes public and internal benchmarks, and some cells are marked unavailable, so cross-system comparison is uneven by row. The failure analysis documents interruptions but not their aggregate cost in training time or compute.

These gaps do not undercut the central result — rising scores with RL steps across four domains for two model sizes, plus transfer to a distilled 9B student and to held-out harnesses — but they define what a reproduction would need to measure. The open-sourced environments, verifiers, framework and mini-harnesses are the paper's own answer to that need, and they are the right starting point for anyone testing whether the three-lever recipe holds outside the reported settings.

Sources used in this answer