Can base models predict post-training coding-agent performance before training?

End-to-end agent tests are too hard for base models

The practical question is which base checkpoint deserves an expensive round of agentic post-training. The obvious test, running the checkpoint end-to-end on an agentic coding benchmark and measuring pass@K, does not work well here: the paper reports that untuned base models fail to operate tool-use harnesses, so pass@K stays near zero. A near-zero score cannot separate a checkpoint with latent coding ability from one without it.

The opposite shortcut is also inadequate. Single-shot or short-horizon coding tasks collapse a multi-step interaction into one fixed prompt and one patch, which avoids tool-calling failures but stops testing the capability that matters: keeping coherent state across many tool-using steps as a repository changes. The paper quantifies the mismatch across its ten base/post-trained pairs: Spearman correlation with post-trained SWE-bench Verified pass@1 ranges from 0.830 on RepoBench XFirst down to −0.394 on HumanEval. Current evaluations are therefore either too agentic to run on base checkpoints or not agentic enough to predict downstream performance.

The decisive step turns a trajectory into a probe

The method starts from a successful trajectory produced by a frontier post-trained model on an agentic coding task, using a harness such as mini-swe-agent. The authors replay that trajectory and rerun the task's tests after every code-changing step. The decisive step T* is the first step whose cumulative patch flips the repository from failing to passing, so the action recorded at that step is certified by the verifier to solve the task given the preceding context.

This construction yields four things per task: the trajectory context up to T*, a golden action, distractor actions that were rejected by the same verifier, and a restorable environment state with an executable verifier. Because the base checkpoint is handed the real harness context rather than being asked to drive the harness from a cold start, the probe isolates whether the checkpoint assigns support to a behavior that post-training could later select. The authors motivate this with a coverage principle: a promising base checkpoint should retain probability mass on behaviors post-training may sharpen, not merely maximize average correctness or conform to today's agent protocol.

Three screens ask three different questions

Decisive-Action BPB measures the byte-normalized negative log-likelihood the base checkpoint assigns to the golden action, so lower BPB means more probability mass on the certified action. Bits per byte normalizes for sequence length and keeps checkpoints with different tokenizers comparable. Patch MCQ is a multiple-choice test: the checkpoint must pick the verifier-certified golden action over alternatives that the same verifier rejected, which measures near-miss discrimination rather than raw likelihood. Prefix-conditioned pass@K is generative: it samples K continuations from the decisive-step prefix and credits any continuation the task's own tests accept, so it measures coverage rather than the probability of one recorded action.

These are not interchangeable metrics. A checkpoint can assign high mass to the golden action yet fail to sample other working edits, or choose correctly among options yet generate poorly. Holding the agentic context and success criterion fixed while varying the question is what lets the three screens be read together.

All three screens track post-trained SWE-bench rankings

The evaluation covers ten base checkpoints, including Nemotron-3-Nano, Nemotron-3-Super, Nemotron-3-Ultra, Qwen-3.5-35B-A3B-Base, DeepSeek-V4-Flash-Base, Kimi-K2-Base, GLM-4.5-Air-Base, Hy3-Preview-Base, DeepSeek-V4-Pro-Base and Gemma-4-26B-A4B. Each probe's per-checkpoint score is correlated with the downstream pass@1 of that checkpoint's post-trained family, using publicly reported scores where available and running evaluations otherwise. Spearman correlation is emphasized because ordering checkpoints correctly is what checkpoint selection needs.

With probes built from DeepSWE trajectories, Decisive-Action BPB reaches Spearman ρ = 0.964 (Pearson r = 0.965), Patch MCQ ρ = 0.867 (r = 0.915), and prefix-conditioned pass@K at K = 32 reaches ρ = 0.951 (r = 0.917) against post-trained SWE-bench Verified pass@1. Probes built from SWE-bench Verified trajectories instead give ρ = 0.915 for BPB, 0.903 for Patch MCQ and 0.906 for prefix-conditioned pass@K. Matching the source benchmark to the target does not buy better rank agreement: the held-out DeepSWE source is closer for BPB (0.964 versus 0.915) and prefix-conditioned pass@K (0.951 versus 0.906), and only Patch MCQ gains from the match (0.903 versus 0.867).

Bounded code benchmarks lose the ranking signal

The paper's comparison against conventional code benchmarks is the sharpest argument for its approach. On the same ten checkpoints, synthesis benchmarks carry almost no rank signal against post-trained SWE-bench Verified performance: MBPP reaches ρ = 0.091 and HumanEval ρ = −0.394. The strongest baseline, RepoBench XFirst at ρ = 0.830, still trails all three DeepSWE-derived probes, which span ρ of 0.867 to 0.964.

The comparison matters because these bounded benchmarks are cheap and widely used. The result says that a checkpoint's score on isolated function synthesis or completion-style repository tasks does not reliably order checkpoints by the agentic capability that post-training will realize. The probes cost more to construct, since they require successful trajectories and a working verifier, but they recover an ordering the cheaper tests miss.

Rank agreement holds on a multilingual target

The DeepSWE-derived probes are also tested against post-trained SWE-bench Multilingual performance. Rank agreement holds or improves: ρ = 0.976 for Decisive-Action BPB, 0.927 for Patch MCQ and 0.945 for prefix-conditioned pass@K. The BPB figure is the highest of the nine probe–target pairs reported across the SWE-bench Verified, SWE-bench Multilingual and Terminal-Bench analyses, and Patch MCQ agrees more closely here (0.927) than with SWE-bench Verified (0.867). The authors suggest one plausible contributor: the DeepSWE source corpus is itself multilingual.

Pearson r is noticeably lower on this target, 0.823 to 0.854, and the paper attributes the gap between the two statistics to a single checkpoint: Gemma 4 26B scores 17.3% on SWE-bench Multilingual against 52.0% on SWE-bench Verified. That outlier compresses linear agreement while leaving the ordering of checkpoints largely intact, which is why the rank statistic is the relevant one for selection.

What the screens can decide before training

The framework's transferable claim is narrower than a general predictor of post-training outcomes. It ranks a cohort of ten public base checkpoints in close agreement with post-trained SWE-bench Verified pass@1, and the authors present it as a way to convert current and future agentic coding benchmarks into base-model evaluations, since the method needs only a benchmark's successful trajectories and its verifier. It does not establish absolute post-trained scores, and the reported correlations come from this ten-checkpoint cohort rather than a larger or differently composed one.

The practical priority follows from the comparison rather than from the correlations alone. When the choice is which base checkpoint to spend post-training compute on, a bounded benchmark such as HumanEval or MBPP can order checkpoints in the wrong direction, while a decisive-step probe built from held-out agentic trajectories preserves the ordering. Decisive-Action BPB is the strongest single screen in the reported pairs, reaching ρ = 0.964 on held-out DeepSWE trajectories and 0.976 against SWE-bench Multilingual, and it requires only the likelihood of one certified action rather than sampling or multiple-choice construction.

Sources used in this answer