Training SLMs on a Budget: Why Mixed Data Complexity is the Secret to RLVR Efficiency
Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes
This paper introduces a systematic study of Reinforcement Learning with Verifiable Rewards (RLVR) in low-data and compute-constrained regimes using Small Language Models (SLMs). By utilizing three novel procedurally generated datasets (Counting, Graph, and Spatial Reasoning), the authors demonstrate that training on mixed-difficulty data achieves up to 5x greater sample efficiency compared to easy-only data.
TL;DR
The prevailing "Scaling Law" wisdom suggests that more data and more compute always lead to better models. However, new research "Learning From Less" proves that in resource-constrained environments, dataset composition beats volume. By utilizing procedurally generated datasets and mixed-difficulty samples, the authors achieved 5x sample efficiency in training Small Language Models (SLMs) using Reinforcement Learning with Verifiable Rewards (RLVR).
The "Verified" Reasoning Frontier
The recent breakthrough of models like DeepSeek-R1 has highlighted the power of RLVR—a method where models are rewarded based on objective, verifiable outcomes (like a math answer or a passing code test) rather than subjective human preferences. While effective, RLVR usually requires massive datasets (e.g., 100k+ samples).
This paper asks a critical question for the rest of us: Can we get the same reasoning gains with just 100 samples and a few A100 GPUs?
The Problem: The High Cost of "Easy" Scaling
Standard fine-tuning often relies on "Easy" data to stabilize training. However, the authors found two major pitfalls:
- Data Homogeneity: Models trained only on easy tasks fail to generalize to the "hard" edge of the distribution.
- Fixed-Budget Trap: In a fixed-compute setting, larger datasets mean fewer optimization updates per example, leading to a performance drop-off—an "Inverted-U" scaling curve.
Methodology: Procedural Generation & Mixed Complexity
To isolate the effects of data, the researchers built three procedural engines:
- Counting Problems: Multi-step numerical logic.
- Graph Reasoning: Finding cliques, paths, and vertex covers.
- Spatial Reasoning: 2D grid movement and relative orientation.
The Secret Sauce: Mixed Tiers
Instead of just "more data," they curated a Mixed-Difficulty set:
- Easy: Solved by ~70-100% of baseline models.
- Medium: Solved by ~34-66%.
- Hard: Solved by 0-33%.
They trained a Qwen3-4B model using Group Relative Policy Optimization (GRPO). This algorithm generates a group of outputs for the same prompt and rewards the model based on the relative advantage within that group, significantly reducing variance in low-data settings.
Key Insights: Diversity is the Ultimate Optimizer
1. The 5x Efficiency Gain
In the Counting Problems task, the model trained on 100 mixed samples performed as well as the model trained on 500 easy samples. Heterogeneity acts as a regularizer, forcing the model to learn robust reasoning paths rather than memorizing easy patterns.
2. The Stability Threshold
The researchers observed that training on very small "Easy" datasets (100 samples) was actually unstable, leading to gradient spikes. Paradoxically, adding "Hard" samples stabilized the training, suggesting that difficulty diversity provides a more informative gradient signal.
3. The Token Budget Bottleneck
In Graph Reasoning, the "Hard" problems often failed because the models exhausted their token limit before finishing the reasoning chain (Chain of Thought). This highlights that for complex reasoning, inference compute (length) is a more rigid constraint than data size.
Critical Analysis & Takeaways
The core takeaway for AI engineers is clear: Don't just scrape more data; curate harder data.
Strengths:
- Proves that SLMs (Small Language Models) can develop significant reasoning capabilities without "Big Tech" compute.
- Highlights the "Inverted-U" risk: if your compute budget is fixed, adding more data can actually hurt your final model accuracy.
Limitations:
- The study is limited to a 4B parameter model; it’s unclear if 70B+ models follow the same 5x efficiency rule.
- Procedural data is "cleaner" than real-world messy data, which might overstate the ease of verification.
Future Outlook
This work paves the way for "Budget-Aware Scaling Laws." In the future, we likely won't just report "samples used," but rather a "Diversity/Compute" ratio that determines the optimal path to high-reasoning SlMs.