Prescriptive Scaling Laws: Why Data Scarcity Demands Bigger Models, Not More Epochs

TL;DR

As we approach the "Data Wall," the industry's default response has been to repeat datasets for multiple epochs. This paper proves that current scaling laws (like Chinchilla) are blind to the "superlinear damage" of overfitting in this regime. By introducing an additive overfitting penalty, the authors reveal a counter-intuitive truth: when data is scarce, you should stop repeating tokens and start building larger models.

The "Blind Spot" in Existing Laws

The Chinchilla law was designed for a world of infinite data. When Niklas Muennighoff et al. (2023) extended it for data repetition, they used an "effective data" count—essentially saying that the 2nd or 3rd time you see a token, it's just "less useful."

The Cornell team argues this is fundamentally wrong. Data repetition doesn't just offer diminishing returns; it actively causes overfitting damage. Furthermore, larger models are more "hungry" and "aggressive"—they overfit on repeated data much faster than small models. Prior laws failed to capture this specific interaction between model size ($N$) and repetition ($R_D$).

Methodology: The Additive Overfitting Penalty

Instead of modifying the token count $D$, the authors add a penalty term to the Chinchilla equation. The intuition is elegant:

1. Tokens are still useful: Even a repeated token helps the model refine its understanding (reducing the "data cost").
2. Overfitting is a separate tax: Repeating data incurs a penalty that scales with how over-parameterized the model is relative to the unique data.

The Complexity Ladder

The authors propose a 4-parameter form that captures the superlinear nature of repetition damage: $L(N,U_D,R_D)=E+\frac{A}{N^{\alpha }}+\frac{B}{(U_D\cdot (1+R_D){)}^{\beta }}+P\cdot R_D^{\delta }\cdot {\left(\frac{N}{U_D^{\gamma }}\right)}^{\kappa }$

Figure: The residual loss grows superlinearly with repetitions ($R_D$), especially for larger models ($N$) on smaller data budgets ($U_D$).

Visual Proof: The Allocation Frontier

The most striking finding is the Compute-Optimal Allocation Frontier. In the Chinchilla world, the line always moves "forward" (more data). In the Cornell model, the frontier actually turns back.

Figure: Our law (blue) shows that at high compute with fixed data, you must scale model size and actually REDUCE the number of epochs to avoid the overfitting cliff.

Case Study: Weight Decay as a "Data Extender"

The authors used their law to investigate Weight Decay ($\lambda$). They found that increasing $\lambda$ from 0.1 to 1.0:

• Incurs a slight "tax" in the single-epoch regime.
• Reduces the overfitting coefficient ($P$) by 70%.

This means strong regularization allows you to train for more epochs before the overfitting penalty destroys performance. Their law accurately predicted the "crossover point" where a highly regularized model eventually beats a standard one in data-constrained settings.

Critical Insights & Conclusion

Takeaways

• Stop the Repetition: There is a mathematical limit to how many times you should show a model the same data. Beyond that, more FLOPs should go into width/depth.
• Regularization is Key: If you must repeat data, you must increase weight decay significantly—often by an order of magnitude.

Limitations

The study focused on models up to 1B parameters. While the power laws are historically robust, the "extremes" of frontier scales (70B+) might involve secondary phenomena like "double descent" which this law does not yet encapsulate.

The Bottom Line: As high-quality human text runs out, the future of AI training isn't just "more compute," but "smarter allocation" that respects the physical reality of overfitting.