How does RSI-Forge turn research papers into environments for self-improvement?

An environment is a problem plus a trustworthy score

Recursive self-improvement, in this paper's framing, is a process in which an agent improves its own capabilities and uses those improvements to guide further changes. That requires more than a question to answer. The agent needs a starting solution it can edit, a way to test changes, and a score that separates real progress from noise or from tricks that raise the number without doing the intended work.

RSI-Forge's unit of output is therefore an executable environment: task instructions, runnable code, code that generates task data, a starting solution, an independently reproduced baseline implementation, and an evaluation configuration. Each environment also splits data into a development set for experimentation and a held-out test set for scoring. The weather-estimation example makes the structure concrete: solvers test changes on 160 development instances, while grading uses 324 held-out instances drawn from different simulation conditions, and the objective is to minimize mean estimation error.

Scores are normalized so the starting solution sits at 0 and a target score at 1. In that same weather task, held-out error is 1.786 for the starting solution and 1.390 for the reproduced paper method, with a target of 1.156, which places the reproduced baseline near 0.629. This normalization is what lets a reader compare progress across tasks whose raw metrics are otherwise unrelated.

Three agents build, reproduce, and review separately

The construction pipeline assigns distinct roles and, importantly, distinct information. An author agent receives the paper and construction requirements and produces the task, the starting solution, and the evaluator. A reproducer agent receives the paper and solver-facing materials and independently implements the paper's method to establish the baseline score. A reviewer agent receives the environment and the reproduction report and either approves it or requests repairs.

The separation matters because the reproducer does not see the author agent's method implementation. That gives the baseline a degree of independence from the task designer, and it lets review assess the environment rather than merely confirm the author's own choices. Failed checks lead to repair or rejection before an environment is accepted.

Construction is not a single clean pass. Of the accepted environments, 79% required at least one revision after failing a construction check, and the reported funnel counts passage after any permitted repairs. The most common defects found by coding repair passages were checks unable to detect the intended failure (66% of repair passages), incorrect or incomplete instructions (63%), checks inspecting a different copy of the code (54%), and score gains without performing the intended work (37%); categories overlap. Construction used a median of 170 million model tokens per accepted environment.

From 10,800 preprints to 210 environments

The scale claim needs its denominators kept straight. Screening covered 10,800 preprints, of which 2,117 (20%) were eligible for construction. Eligibility required properties such as a measured baseline, a metric computable by a deterministic script, a method fitting the available compute budget, measurable room for improvement, and a licence permitting redistribution; all source papers are from 2026.

Only 213 of those 2,117 eligible papers entered the main construction run, so the funnel counts do not estimate the yield from all eligible papers. In that run, 213 attempts began, 199 passed design, 198 passed build, 189 passed independent reproduction, 180 passed end-to-end testing, and 180 entered the main corpus. The main run used Claude Opus 5 for all three agent roles, with separate sessions and role-specific inputs.

A second run replaced the construction model with GPT-5.6-sol and built 30 additional environments using the same queue and checks, bringing the total to 210 environments across 18 fields. The two sets are unpaired by source paper and field: the second run tests whether the procedure works with another backend, not how much one model beats another. The 180 main-run environments include 6 to 14 per field, extending beyond machine learning into robotics, computational physics, statistics, and software engineering.

Experts agree on headroom but not on depth

Quality was rated on a shared nine-dimension rubric with scores from 1 to 5. Three agent judges rated all 180 main-run environments, and human domain experts rated 90 of them, one expert per environment, each holding a doctorate in the relevant field. On that shared subset, the mean across environments and dimensions was 4.24 for experts and 4.54 for the three-judge panel.

The two groups converge on the properties needed to measure progress. Experts rated room above the starting solution highest at 4.81 and score discrimination between attempts at 4.74. They were more critical elsewhere: resistance to saturation received 3.09 from experts versus 3.98 from the agent panel, faithfulness to the source paper 3.72 versus 4.22, and shortcut resistance 4.09 versus 4.67.

The saturation gap is the widest and has a concrete illustration. One expert gave the lowest rating after a supplied search method covered about 69% of the declared improvement band in roughly five CPU-minutes, and also cited solver results reaching the target score in the first session, which agent judges did not receive. The paper notes that reaching a normalization target does not rule out further raw gains or establish that the research problem is exhausted. The comparison also reflects unequal evidence: only experts received summaries of solver scores over successive sessions, and some experts cited those summaries.

A faithfulness example shows what the lower ratings capture. One expert assigned 2 against the panel's 4.5 because a task kept 50-way zero-shot classification and comparison of audio and text semantics but replaced real THINGS-EEG recordings, pretrained feature pipelines, and the trimodal BraVL VAE with synthetic low-dimensional arrays and a linear-Gaussian brain model. Adapting a problem to a bounded compute budget can preserve the broad question while omitting substantive parts of the source method.

Models improved, but mostly in the first attempt

The repeated-improvement test used 120 environments sampled from the main construction run and four models, each given 3 successive attempts. Each attempt inherited prior code and notes while model weights stayed fixed, so the setup measures progress within a task rather than learning that changes the model. At least one model improved after the first attempt in 84% of environments, and models outperformed the reproduced paper methods in 68 of the 120 environments.

Those gains are not merely parameter tuning. Transcript analysis identified work beyond parameter tuning in 95% of the successful attempts that beat the reproduced baselines. Median final normalized scores on the 120 environments were 0.80 for Opus 5, 0.76 for GPT-5.6-sol, 0.65 for DeepSeek, and 0.33 for Haiku 4.5, on the scale where the starting solution is 0 and the target is 1.

The paper also reports that gains are concentrated in the first session for the stronger models. It explicitly does not treat diminishing gains as proof of task saturation, since they may instead reflect limits in how agents search for alternatives or use inherited records. The study also does not isolate the effect of inherited records from the effect of extra solving time.

Lower scorers explore less and accept noise

Trajectory analysis describes how weaker performance looked in practice. Models scoring lower explored less, more often accepted gains smaller than the reported standard error, and relied more heavily on tuning to the development set. Across models, some final submissions were worse than previously saved versions, meaning that finding a better solution did not guarantee it was retained.

The paper is careful about what these patterns mean. It presents them as associations, not causes: the study does not establish which practices cause higher performance. It frames experimental judgment and final selection as concrete targets for future interventions, and the saved programs, measurements, and handover records as the basis for evaluating such interventions.

This is where the environment design and the behavioral findings connect. Because each environment records the sequence of changes an agent makes, the same artifacts that score a solution also expose whether an agent tested enough alternatives, whether it distinguished a real gain from measurement noise, and whether it kept its best result.

What these environments do not show

Several boundaries qualify the headline numbers. Tasks are adapted to bounded compute, so surpassing a reproduced baseline does not establish an advance on the original research problem. No trajectories were used for training, so the study does not establish broader gains in research capability. Some environments left held-out grading materials accessible during solver runs, and one expert's proposed attack depended on runtime access controls rather than task wording alone.

The review evidence is also partial. Expert review covers 90 of the 180 main-run environments with one expert each and no human inter-rater estimate, and model-generated trajectory labels lack human validation. The construction roles were not ablated, so the individual contribution of the author, reproducer, and reviewer agents remains unresolved. Access restrictions limit independent replication of the full corpus.

The comparison with manually constructed tasks is unpaired. Under the same rubric, agent judges rated RSI-Forge environments higher than 34 manually constructed RSI-Exam tasks on score discrimination (4.65 versus 2.64) and development-test distinction (4.67 versus 3.23), while RSI-Exam scored higher on resistance to saturation (4.41 versus 3.98). Because the task sets are unpaired, these differences do not isolate the effect of automated construction.

The bottleneck is depth, not headroom

The most decision-relevant split in this paper is between what the pipeline reliably produces and what it struggles to preserve. Experts gave their highest ratings to room above the starting solution (4.81) and score discrimination (4.74), and models did improve and beat reproduced baselines in 68 of 120 tested environments. On the properties that decide whether an environment can sustain repeated improvement, the picture is weaker: resistance to saturation at 3.09 and faithfulness to the source paper at 3.72 from experts.

That gap points to a specific priority rather than a general caution. If the goal is environments that reward several independent ideas instead of one change that captures most of the range, the binding constraint is preserving the depth of a research problem while making it executable under a fixed compute budget, and the paper argues that targeted expert review remains valuable for assessing exactly these properties as construction scales.

The solver results reinforce where attention is best spent. Gains concentrated in the first session, lower-scoring models accepting changes smaller than the reported standard error, and final submissions worse than previously saved versions all concern how agents conduct and retain experiments. For readers judging whether automated environment construction is ready, the useful threshold is not whether tasks can be generated at scale, which this paper demonstrates across 210 environments in 18 fields, but whether the resulting tasks resist saturation and shortcutting well enough to keep measuring genuine progress.

Sources used in this answer