How can recursive data synthesis escape the RLVR zero-advantage trap?

Why 88% of a large math corpus teaches nothing

Reinforcement learning with verifiable rewards (RLVR) trains a model by sampling several rollouts per problem and rewarding those that reach a checkable correct answer. The learning signal comes from variation inside that group: if some rollouts succeed and some fail, the policy can be pushed toward the successful ones. The paper calls the failure mode the zero-advantage trap: when every rollout succeeds or every rollout fails, the group-relative advantage is zero and the problem contributes no gradient.

The authors measured this on a corpus of about 78,000 math problems pooled from DAPO-Math-17K, Skywork-OR1-RL-Data, Turing Open-RL and a subset of StackMathQA. Sampling N=8 responses from gpt-oss-120b at a 32K rollout budget, only 12.2% of problems fell in the observed Goldilocks band of mixed outcomes; 6.3% were never solved, and the rest were solved in every rollout. Roughly 88% of the corpus therefore sat in the trap.

The unit matters here. The 88% figure is not a property of the problems alone: it is defined for a fixed target policy (gpt-oss-120b), a fixed number of rollouts (8), and a fixed budget (32K tokens). The paper reports that 32.3% of problems change class when the policy changes, so the same corpus can look very different to a different model.

Filtering concentrates signal but cannot create it

A natural response is to keep only the mixed-outcome problems. The paper tests this by building a roughly 7K-problem proxy whose hard, Goldilocks and stale proportions match the full corpus, then extracting about 862 Goldilocks problems and a stricter 696-problem subset that also admits an in-budget correct solution under a 16K training limit. Both subsets beat the full-mixture proxy while using an order of magnitude fewer problems, so filtering does concentrate useful signal.

But filtering is subtractive. It can only select problems the corpus already contains, and it discards the hard problems the policy never solves. In the full-corpus comparison, the best filtered real-data subset improved gpt-oss-120b by +5.8 points, while training on the entire 78K source corpus improved it by only +1.1. Filtering recovers part of the gap but cannot manufacture a problem for a capability the corpus does not cover.

Hard seeds specify what to synthesize

The paper's alternative starts from the problems filtering throws away. A hard seed is a problem the target policy fails on every rollout; the authors work with about 1,250 seeds on which gpt-oss-120b obtained 0/8 correct at 16K. The seed's verified teacher reasoning trace is decomposed top-down into subproblems, and a decomposition is kept only if conditioning a student on the child problems and their worked solutions raises that student's empirical accuracy on the parent. This test decides whether the children are useful scaffolds, not whether a final synthetic problem is in the Goldilocks band.

The retained tree is then regrown bottom-up: harder new leaves are synthesized and recursively composed into parallel trees, and the resulting root problems are kept only if they produce mixed target-policy rollouts and at least one normally terminated correct response within the budget. The seed supplies the reasoning operations and their composition; the policy and budget decide which descendants are useful.

The paper reports that this separation of roles is what direct prompting lacks. Asking a teacher to vary a hard problem directly produced shallow variants: in a blinded judge study over 300 math and 100 finance problems, seed synthesis often kept recognizable techniques but through paraphrase, contextual rewrite or numerical substitution rather than a substantively new problem. On the same 64 seeds, direct synthesis reached an all-three candidate (blueprint-preserving, novel, Goldilocks) for 13 seeds, versus 28 for MagicBeanStalk, even when direct synthesis was given a matched construction budget of 16.0M estimated tokens against MagicBeanStalk's 14.3M.

Gains on eight math benchmarks and their budget dependence

Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems raised gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4 Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. The comparison conditions are the point: the full 78K source corpus gave +1.1, the best filtered real-data subset gave +5.8, and Goldilocks-filtered seed synthesis gave +5.6. The synthetic data beats both the raw corpus and the best real subset the authors could select.

The advantage is budget-dependent. Evaluating the 16K-trained gpt-oss-120b checkpoints at larger inference budgets, MagicBeanStalk's gain over base shrinks from +8.6 at 8K to +8.3 at 16K, +1.0 at 24K and +1.5 at 32K, while the filtered real subset turns negative at 24K and 32K. On the harder MathArena Apex-Shortlist, the MagicBeanStalk checkpoint gains +11.0 at 16K, +13.3 at 24K and +10.9 at 32K over base. The paper's own framing is that usefulness is relative to the policy and the rollout budget, so these numbers should be read as gains for a specific training budget, not as a permanent capability jump.

The synthetic training also recovers the original hard seeds

A separate check asks whether training on descendants transfers back to the parents. The authors reevaluate the roughly 1,250 source seeds on which the initial gpt-oss-120b sample scored 0/8 at 16K. A fresh base sample recovers 9.5% of these seeds, and the paper reports that MagicBeanStalk training improves recovery beyond that baseline, which is the direct evidence that the synthesized problems target the reasoning the hard seeds required rather than unrelated material.

This is the mechanism claim the paper makes: because the synthetic problems preserve the seed's reasoning blueprint, learning them should help on the seed itself. The blinded judge study supports the blueprint-preservation part, and the seed-recovery result supports the transfer part, but both are measured on the authors' own seed population and evaluation protocol.

Finance transfer from 340 expert questions

The paper also tests whether the recipe leaves math. From only 340 expert finance questions, it synthesizes over 14,000 verifiable problems and improves Qwen3 by 6.9 points on SecQue and 2.4 points on FinanceBench over base. The seed-to-problem expansion ratio is far larger than in math, which is consistent with the method's premise that a small set of hard, well-structured seeds can anchor a much larger calibrated set.

The finance result is narrower evidence than the math result: two benchmarks, one base model, and a domain where the authors explicitly note that further testing would be needed before relying on synthetic data to train models for financial decisions. It shows the pipeline is not math-specific, not that the gains generalize across finance tasks.

What the evidence does not settle

Several limits sit close to the results. The synthesis pipeline relies on teacher models, some of them proprietary APIs whose updates the authors do not control, so exact reproduction from scratch may differ. Synthetic problems can contain errors; the authors audit a sample with three additional models and report that at least one judge verifies 89.3% of math and 100% of finance problems, while all three reject 8.0% of math and 0.0% of finance. The downstream training-dynamics comparison is a separate control run over one epoch on a fixed seed population, not the full-budget results.

The zero-advantage characterization is also policy- and budget-relative by construction. The 88% figure, the Goldilocks band and the reported gains are all measured against specific policies, rollout counts and token budgets, and the paper itself shows that changing the policy reclassifies about a third of problems. A reader should treat the method as a way to generate data for a named policy and budget, not as a claim that any large corpus is intrinsically weak.

Where the priority shifts for RLVR data work

The paper's most decision-relevant comparison is not synthetic versus real data in general, but calibrated synthetic data versus the best real subset a filter can extract. On gpt-oss-120b at a 16K budget, that is +8.6 versus +5.8, and the gap widens on the harder Apex-Shortlist benchmark. The practical priority it implies is to spend effort on preserving the reasoning blueprint of hard seeds and on calibrating difficulty to the target policy and budget, rather than on enlarging the corpus or filtering it harder.

The threshold to watch is the Goldilocks band itself: a problem is useful only when some rollouts succeed, some fail, and at least one correct solution fits inside the training budget. Problems outside that band, however novel, add nothing to group-relative learning. That condition, rather than corpus size, is the variable this paper puts at the center of RLVR data scaling.

Sources used in this answer