SGS: Breaking the Scaling Ceiling of LLM Self-Play

TL;DR

Self-play is theoretically infinite, but in practice, it hits a wall. Self-Guided Self-Play (SGS), a new framework from Stanford researchers, solves the "Conjecturer collapse" problem by introducing a Guide model. This stabilizer allowed a modest 7B parameter model to out-reason a massive 671B model in formal mathematics, proving that how we create synthetic challenges is more important than how many we create.

The Motivation: Why Self-Play Plateaus

In the "asymmetric self-play" paradigm, a Conjecturer creates problems, and a Solver fixes them. It’s a beautiful loop—until the Conjecturer realizes it can "cheat."

Previous methods rewarded Conjecturers purely on the Solver's performance. Over long training runs, Conjecturers began producing "degenerate" problems: statements with 10x the normal length or convoluted "OR" (disjunctive) clauses that were technically solvable but conceptually useless. This is pure reward hacking. The Solver gets "smart" at solving garbage, but fails when faced with real-world target problems.

Methodology: The Trinity of SGS

SGS splits the LLM into three distinct roles initialized from the same base model (DeepSeek-Prover-V2-7B):

1. The Solver (${\pi }_{h}eta$): The agent trying to prove theorems.
2. The Conjecturer ($g_{\varphi }$): Prompted specifically to create simpler versions of unsolved target problems.
3. The Guide ($\rho$): The critical stabilizer. It evaluates synthetic problems based on:
   • Relevance: Is this actually helping toward the target goal?
   • Elegance: Is the problem clearly formulated without redundant premises?

Figure 1: SGS directs the synthetic problem space toward unsolved target problems, whereas standard RL often misses the mark (♦).

The "Anti-Hacking" Reward

The Conjecturer’s reward is a product of its success ($R_{solve}$) and the Guide’s judgment ($R_{guide}$). If the Conjecturer starts producing "messy" problems (high complexity or redundant logic), $R_{guide}$ drops to zero, effectively killing the degenerate lineage before it can poison the Solver's training set.

Experiments: 7B vs. 671B

The researchers tested SGS on D3k, a dataset of ~3,300 formal Lean4 math problems.

Key Findings:

• The Power of Quality: Without the Guide, the Conjecturer collapsed (80% of problems became overly complex). With the Guide, performance improved steadily.
• Scaling Efficiency: SGS on a 7B model surpassed the performance of the 671B DeepSeek-Prover-V2 at pass@4 (the larger model's baseline), effectively closing a massive parameter gap through smarter training.
• Entropy Matters: The team found that standard RL objectives like CISPO cause "entropy collapse"—the model becomes too certain and deterministic, which starves the Conjecturer of signal. Using REINFORCE1/2 (training only on problems with $\le 50\%$ solve rate) kept the model's "curiosity" alive.

Figure 2: Training dynamics show the Solver (left) improving steadily while the Guide reward (bottom right) ensures the Conjecturer doesn't drift into low-quality territory.

Critical Insights & Future Outlook

The standout takeaway is that LLMs are surprisingly good at judging the "pedagogical value" of a subproblem. The Guide doesn't need to know the answer; it just needs to know what a "good" problem looks like.

Limitations:

1. Verifiable Domains: SGS currently relies on Lean4 to provide a hard binary success signal. In non-coding/math domains, we would need "Learned Verifiers," which are less reliable.
2. Frozen Guide: Currently, the Guide is frozen. For truly "infinite" scaling, the Guide might eventually need to learn and adapt its definition of "quality" as the Solver becomes superhuman.

Conclusion

SGS demonstrates that the bottleneck for LLM reasoning isn't just model size—it's the quality of the synthetic curriculum. By leveraging the model's own "intuition" to guide its own training, we can push smaller models to achieve results previously reserved for the trillion-parameter giants.