[2026] ReGFT: Breaking the Sparse Reward Wall in Mathematical RL

TL;DR

Reinforcement Learning (RL) is the powerhouse behind models like o1 and R1, but it hits a wall when problems get too hard for the base model to solve even once—if there's no "correct" answer sampled, there's no reward signal. Reference-Guided Fine-tuning (ReGFT) solves this by using human solutions as "scaffolding" to help the model generate its own correct traces for hard problems before RL starts. This creates a high-quality "jumping-off point" that accelerates RL and boosts final performance.

The Sparse Reward Bottleneck

In the current RLVR (Reinforcement Learning with Verifiable Rewards) paradigm, we reward the model if it hits the right answer. However, if you give a high-school level model an International Math Olympiad (IMO) problem, the probability of it stumbling upon the correct proof by sheer luck is effectively zero.

Traditional fixes have failed for two reasons:

1. ReFT (Reinforced Fine-tuning): Only trains on what the model already got right. It doesn't help with problems currently beyond its reach.
2. Direct SFT on Human Solutions: Human proofs often use "leaps of logic" or notations that the model doesn't understand, leading to poor imitation and "brittleness" during inference.

Methodology: The "Partial Hint" Insight

ReGFT introduces a simple but profound shift. For problems the model cannot solve, the authors provide a partial reference solution (e.g., the first 80% of the steps).

Instead of just reading the answer, the model is tasked to:

1. Synthesize the remainder of the proof.
2. Ensure the logic is self-consistent and leads to the correct answer.

By doing this, the resulting training data is correct (verified by the reference) but model-native (written in the model's own style).

Figure 1: While ReFT relies on standard sampling, ReGFT uses reference guidance to recover training signals from "impossible" problems.

Experimental Results: Scaling the Frontier

The authors tested ReGFT using Qwen3-4B on the OmniMath dataset and evaluated it on the brutal AIME and Beyond-AIME benchmarks.

1. Superior RL Initialization

Models started with ReGFT didn't just learn faster; they reached a higher "plateau." As shown in the charts below, ReGFT + DAPO (a state-of-the-art RL algorithm) consistently dominates the baseline RL and ReFT-initialized models.

Figure 2: Training curves across AIME benchmarks. ReGFT (blue) shows higher efficiency and better final accuracy.

2. Robust Test-Time Scaling (Pass@k)

One of the most impressive findings is that ReGFT improves the model's Pass@k scaling. Usually, RL can bias a model to find one specific way to solve a problem (causing saturation). ReGFT, by introducing diverse human-guided paths, allows the model to benefit more from increased "thinking time" (more samples) at inference.

Why This Matters

For the AI community, ReGFT represents a move away from "blind" RL towards a more structured "Guided Evolution." It suggests that the frontier of AI reasoning isn't just pushed by more compute during RL, but by more intelligent ways to utilize the vast amounts of human mathematical knowledge we already possess without forcing the model to act "un-naturally."

Limitations & Future Work

• Verifier Limits: The paper notes that rule-based verifiers still struggle with proof-style answers, leading to false negatives.
• Domain Expansion: While math is naturally verifiable, applying ReGFT to code or logic puzzles remains an exciting next step.

Conclusion

ReGFT proves that human references shouldn't just be used for SFT. By using them as "navigational beacons" to guide the model's self-generation, we can unlock the "Hard" mode of RL and continue pushing the boundaries of what these models can solve.