A predictor that ranks experiments before they run
The paper asks whether a language model can act as a Research World Model (RWM): a single model that, given a research environment and a candidate intervention, predicts the gain that intervention would produce if executed. The environment fixes the reference model, data, configuration, budget and evaluation protocol; the intervention is a concrete change to that starting recipe, such as a different learning-rate schedule or data mixture. The predicted gain is meant to let a researcher or an automated agent spend limited GPU budget only on the most promising candidates.
This framing borrows the world-model idea from control, where a model predicts the consequences of actions so an agent can plan. Here the intervention plays the role of the action and the measured experimental outcome is the consequence. The paper's motivation is a mismatch: AI research agents can propose plausible, implementable experiments faster than those experiments can be executed, so deciding which ones deserve training or evaluation budget requires anticipating their effects.
The evidence base is 2,653 experimental records drawn from nine environments spanning pretraining, post-training and inference, together representing 171,862 H100 GPU-hours. Records are triples of environment, intervention and measured gain, and they form the history a model can condition on. The default RWM backbone is Claude Opus 5 with a 1M context window and HIGH reasoning effort, held fixed when the paper compares knowledge conditions.
Gain error and ranking quality measure different things
Two metrics carry most of the argument and should not be conflated. Mean absolute error (MAE) measures how close predicted gains are to measured gains in the environment's own unit, where lower is better. Spearman correlation measures whether the model orders interventions the same way the experiments did, where higher is better. A model can rank candidates well while being poorly calibrated in absolute terms, and the paper reports both.
Gain itself is defined relative to a reference and depends on the metric. For bits per byte (BPB), gain is the reference score minus the candidate score, so positive means improvement. For accuracy and throughput, gain is the candidate score minus the reference score, and accuracy differences are multiplied by 100 to report percentage points. The paper treats final performance gain as its primary outcome.
A third family of metrics appears once prediction is used for decisions. Regret@3 measures the gain lost by failing to pick the best candidate when selecting three interventions from a pool of 16, roughly a 20% execution budget; NDCG@3 scores the positive gains and ordering of the selected three. These selection metrics answer a different question from global ranking, because a good overall ordering need not identify the few highest-gain candidates.
Real experiment records sharpen within-environment predictions
The paper compares a no-knowledge setting (W0, with an empty history) against a record-conditioned setting (Win-env) that receives records from the environment being evaluated. Across all five environments tested this way, records improved both gain prediction and intervention ranking, raising mean Spearman correlation by 0.27 over W0.
The per-environment numbers show the range rather than a single effect. On OLMo3-100M, Spearman rose from 0.62 to 0.90 and MAE fell from 2.93e-2 to 2.03e-2 BPB. On inference optimization, Spearman rose from 0.54 to 0.88 and MAE dropped from 13.19 to 1.34 tokens/s. Math distillation moved from 0.43 to 0.70 Spearman, and Code RL from 0.48 to 0.62.
The record-conditioned RWM also beat two feature-based baselines, k-nearest-neighbor regression and ridge regression, on both metrics in every environment, even though those baselines received the same records and candidate inputs. The paper reads this as evidence that a language model can learn more effectively from experimental experience than a standard regressor over the same inputs.
Pretraining experience transfers to a new environment
Cross-environment evaluation takes each of five autoregressive pretraining environments as a target in turn. The model receives 244 records from the other four environments and predicts gains for 71 interventions in the target, with no target-environment records and with held-out interventions and near-duplicate groups excluded from the source records.
Source records improved Spearman point estimates in all five targets, with a macro-average of 0.73 against 0.63 for W0 and 0.36 for kNN. The headline transfer result is the Qwen3 target: using only pretraining experience from OLMo3, Marin and Nanochat, the RWM reduced selection regret in Qwen3 by 78% compared with the zero-experience setting.
The paper also shows where naive transfer breaks. kNN beat W0 on four within-environment panels but fell below it on every transfer target, because semantically similar changes can mean different modifications under a different starting recipe. In one case, a Muon optimizer variant looked beneficial by source analogues, but the measured gain on Nanochat, which already uses Muon, was −0.036 BPB; W0 predicted −0.035. Aggregate transfer gains therefore coexist with individual cases where source analogues mislead.
Better rankings become better experiment selections
To test whether prediction helps decisions, the paper simulates a fixed execution budget by selecting three interventions from each pool of 16 candidates, averaged over 200 random partitions of each environment's test panel. Random selection provides the reference expected score.
Within-environment knowledge reduced Regret@3 substantially: the equal-weight mean of per-environment relative reductions from W0 to Win-env was 58.2%, and NDCG@3 improved in every environment. Cross-environment knowledge reduced macro-average Regret@3 by 52.2% relative to W0, with both selection metrics improving in all targets except Marin. Both feature-based baselines had higher macro-average Regret@3 than W0 in the cross-environment setting.
The paper also prices the prediction step. Scoring 16 candidates with one call each costs about $6.3 at the observed mean API cost, while the median environment's average compute cost per experiment is 42.3 H100 GPU-hours, or roughly $127 at $3 per GPU-hour. Scoring a full candidate pool therefore costs about 5% of one representative experiment, which is the comparison that makes prediction worth doing under a tight budget.
Multi-round search gains from both kinds of knowledge
The multi-round Autoresearch test runs five evaluations on OLMo3-100M, each with eight rounds. Sixteen new candidate interventions arrive each round, each method selects three from those currently available, and the total selection budget is 24 per evaluation. All methods see the same candidates and face the same budget.
Relative to the no-knowledge setting, RWMs with within-environment knowledge increased the best gain achieved by 15.8%, and RWMs with cross-environment knowledge increased it by 11.6%. The same cross-environment records were also given to kNN and ridge for comparison. These are gains in the best outcome found under a fixed budget, not improvements in prediction accuracy, so they measure whether forecasting actually changes what gets discovered.
Experimental knowledge outweighs model choice and reasoning effort
Ablations across 13 language models used as RWM backbones show that adding research knowledge improves intervention ranking more than changing models or increasing reasoning effort alone. On the OLMo3-100M panel, the two performance ranges do not overlap: the lowest-performing backbone with records exceeds the highest-performing backbone without them.
The reasoning-effort comparison points the same way. Even the LOW effort setting with records outperformed every effort setting without records, including XHIGH. The paper's interpretation is that changing the backbone alone does not compensate for missing experimental evidence, which is why it argues for accumulating experimental records rather than only scaling the predictor.
Where the transfer claim stops
The demonstrated benefits arise through in-context learning: records are supplied to the model at prediction time rather than used to train it. Cross-environment evidence is currently limited to autoregressive pretraining, so the 78% Qwen3 regret reduction and the 52.2% macro regret reduction should not be read as established for post-training or inference environments, where only within-environment results are reported.
The paper also notes that constructing the records and evaluating RWMs consumes substantial compute, and it reports the 171,862 H100 GPU-hours so efficiency claims can be weighed against that cost. Language-model predictions depend on a commercial API and are not bitwise reproducible, though the model version, context window and reasoning effort are fixed.
The practical priority the evidence supports is a comparison, not a blanket recommendation: under a fixed selection budget, supplying real experimental records to a language-model predictor changed selection outcomes more than swapping the backbone or raising reasoning effort, and cross-environment records delivered most of that benefit without any target-environment data. The open question the authors flag is whether training RWMs on diverse records generalizes better than in-context learning in unseen environments, and how to balance immediate improvement against acquiring knowledge for future decisions.
Sources used in this answer
Language Models as AI Research World Models
Language models given real experimental records predict AI research intervention outcomes better within and across environments, reducing selection regret by up to 78% and improving multi-round Autoresearch best gain by 15.8% with in-environment knowledge.