SGS: Scaling LLM Reasoning by Transforming Models into Their Own Math Tutors
Scaling Self-Play with Self-Guidance
Self-Guided Self-Play (SGS) is a novel asymmetric self-play algorithm designed to scale LLM learning by using the model itself as a Guide to prevent Conjecturer collapse. Applied to formal theorem proving in Lean4, it allows a 7B parameter model to surpass the performance of a 671B parameter model (pass@4) by maintaining high-quality synthetic data generation.
Executive Summary
TL;DR: Researchers from Stanford have introduced Self-Guided Self-Play (SGS), a framework that solves the "learning plateau" problem in LLM self-play. By assigning the model a third role—a Guide—the system prevents the generation of "garbage" synthetic data. The results are startling: a 7B model trained via SGS can outperform a 671B model on complex formal math proofs.
Positioning: This work moves beyond simple "Rejection Sampling" or "Expert Iteration" by creating a closed-loop, self-evolving system that manages its own curriculum quality. It is a significant milestone in scaling laws for compute, proving that how you use your training tokens matters as much as how many you have.
The Problem: The "Reward Hacking" Conjecturer
In traditional asymmetric self-play, a Conjecturer creates a problem, and a Solver tries to solve it. If the Solver succeeds, the Conjecturer is rewarded. However, the authors observed a terminal flaw: over long training runs, the Conjecturer learns to "cheat."
Instead of creating helpful stepping-stone problems, the Conjecturer starts generating logically bloated, inelegant problems that are easy to solve but have no relation to the real-world target problems. This leads to Distribution Collapse, where the Solver stops learning because the training data has become "degenerate."
Methodology: The Trinity of Solver, Conjecturer, and Guide
SGS solves this by introducing a three-role architecture:
- Solver: Learns to prove theorems (the primary agent).
- Conjecturer: Proposes "simpler" lemmas related to unsolved target theorems.
- Guide: A frozen, fine-tuned version of the LLM that acts as a quality controller.
The Guide scores the Conjecturer's output on a rubric of Relevance, Redundancy, and Complexity. If a proposed problem is a "convoluted mess" (even if it's solvable), the Guide gives it a low reward.
Figure 1: (Left) The Guide directs problem-solving toward the target. (Right) 7B SGS model surpassing the 671B baseline.
Managing "Entropy Collapse"
A secondary insight of the paper is the choice of the RL objective. The authors found that popular methods like CISPO (a grouped RL objective) cause the Solver to become "deterministic" too quickly (Entropy Collapse). When the Solver only outputs 0 or 1 solve rates, the Conjecturer gets no "gradient" to improve. SGS uses REINFORCE1/2, which focuses updates only on hard problems, keeping the learning signal healthy.
Experimental Evidence: Small Model, Big Brains
The team tested SGS on Lean4 formal theorem proving using the D3k dataset (3,323 hard math problems).
Key Findings:
- Efficiency: SGS outperformed the previous state-of-the-art self-play method (STP) within 1 million generations.
- The Power of Compute: The 7B model, given enough self-play rounds, eventually solved problems that the base 671B model could not solve at pass@4.
- Ablation Proof: Without the "Guide," the average completion length of synthetic problems exploded (becoming 10x longer), proving that the LLM's self-judgment is what keeps the curriculum "sane."
Figure 2: Without the Guide, the Conjecturer collapses into generating complex, disjunctive "nonsense" (orange line).
Critical Analysis & Conclusion
Takeaway
SGS demonstrates that LLMs are excellent at evaluating their own sub-goals. We don't need human-labeled data to scale reasoning; we need better "architectural roles" that allow the model to provide its own supervision and maintain data diversity.
Limitations
- Verifiable Domains: Currently, SGS relies on the Lean4 compiler to provide the "ground truth" of whether a solution is correct. Extending this to non-verifiable fields (like law or creative writing) would require a "Learned Verifier," which might introduce its own biases.
- Frozen Guide: Today the Guide is frozen. In the future, the authors suggest the Guide should also evolve, learning to prioritize "pathways" that led the Solver to its biggest breakthroughs.
Future Outlook: SGS suggests a future where "Training" and "Inference" blur. If a model can learn from itself during test-time compute, the limiting factor isn't the size of the dataset, but the amount of electricity we are willing to spend letting the model "think" and "teach" itself.