Chinchilla: Rethinking Compute-Optimal Scaling for Large Language Models

TL;DR

TL;DR: This paper overturns the prevailing "bigger is better" scaling dogma by showing that, for a fixed compute budget, model size and training data should be scaled equally. The authors train over 400 models to derive a new scaling law, and validate it with Chinchilla — a 70B-parameter model trained on 1.4 trillion tokens — which outperforms the much larger Gopher (280B) on nearly every benchmark, including a 7.6% absolute improvement on MMLU (67.5% vs. 60.0%).

Background Positioning: This is a landmark empirical study that corrects the influential Kaplan et al. (2020) scaling law, which had steered the field toward massive parameter counts at the expense of training data. It is a "theory revision" paper that provides actionable guidance for allocating compute budgets, and has already shaped the training strategy of subsequent LLMs (e.g., Llama, PaLM 2).

Problem & Motivation: The Undertraining Epidemic

Since the introduction of GPT-3 (175B, 300B tokens) and Gopher (280B, 300B tokens), the de facto approach has been to increase model size while keeping the number of training tokens roughly constant (~300B). This was justified by the scaling law from Kaplan et al. (2020), which claimed that for a 10× increase in compute, model size should grow by 5.5× while tokens only by 1.8×. The result: massive overparameterization without commensurate data.

Why was Kaplan et al. wrong?

1. Fixed learning rate schedule: Kaplan used a cosine schedule configured for 130B tokens across all models, meaning models trained for fewer tokens had an overestimated loss (due to premature learning rate decay). This artificially inflated the value of model size over data.
2. Small model bias: The majority of models in Kaplan et al. were under 100M parameters. The authors show that the FLOP-loss frontier exhibits curvature (see Appendix E), so extrapolating from tiny models to 100B+ models leads to incorrect conclusions.
3. Ignoring data scaling: The field was so focused on engineering mega-models that the fundamental question — "Is it better to train a smaller model on more data?" — was never rigorously tested at scale.

The authors' key insight: The loss function can be decomposed as L(N,D) = E + A/N^α + B/D^β, where the last two terms represent approximation error (from limited model size) and optimization error (from limited data). For a fixed FLOP budget (≈6ND), you need to balance these two terms. The optimal scaling exponents are α/(α+β) for N and β/(α+β) for D. Empirically, α ≈ 0.34 and β ≈ 0.28, leading to equal scaling (a≈0.5, b≈0.5).

Methodology: Three Ways to Find the Optimal Frontier

The authors train over 400 models (70M to 16B parameters) on 5B to 500B tokens, using three independent approaches.

Approach 1: Fixed Model Size, Varying Training Steps

For each of several model sizes, train 4 copies with different numbers of training steps (cosine cycle matched to steps). Obtain a continuous loss-vs-FLOPs curve via interpolation, then for each FLOP budget pick the (N, D) pair that gives the lowest loss. Fit a power law: N_opt ∝ C^a, D_opt ∝ C^b → a=0.50, b=0.50.

Approach 2: IsoFLOP Profiles

For fixed FLOP budgets (6×10^18 to 3×10^21), vary model size and plot the final loss. Fit a parabola to find the minimum loss point (best model size). Then fit power laws across budgets → a=0.49, b=0.51.

Approach 3: Parametric Loss Function

Fit the entire loss landscape with L(N,D) = E + A/N^α + B/D^β using Huber loss and L-BFGS. The fitted parameters: α=0.34, β=0.28, E=1.69, A=406.4, B=410.7. From this, the optimal exponents are a=β/(α+β)=0.46, b=α/(α+β)=0.54.

All three approaches converge to a≈0.5, b≈0.5, i.e., doubling compute should double both model size and data. This is in stark contrast to Kaplan et al. (a=0.73, b=0.27).

Figure 1. Overlaid predictions. The three methods all predict that for current compute budgets, models should be much smaller and trained on more tokens.

Figure 4. Parametric loss fit with efficient frontier. The optimal model size for Gopher's FLOP budget is predicted to be ~40B parameters.

Experiments & Results: Chinchilla vs. the Giants

To validate the new scaling law, the authors train Chinchilla — a 70B-parameter model on 1.4 trillion tokens (4× more data than Gopher, same FLOP budget). The architecture is similar to Gopher but uses AdamW, a slightly modified tokenizer, and higher-precision optimizer state.

Language Modelling

On all 20 subsets of The Pile, Chinchilla achieves lower bits-per-byte (bpb) than Gopher. On WikiText-103, perplexity drops from 7.75 (Gopher) to 7.16.

Figure 5. Chinchilla consistently outperforms Gopher across all Pile subsets.

MMLU (Massive Multitask Language Understanding)

Chinchilla achieves 67.5% average 5-shot accuracy, compared to Gopher's 60.0% — a 7.6% absolute improvement. It even surpasses the expert forecast for June 2023 (63.4%). On 4 individual tasks, it exceeds 90% accuracy (high_school_gov_and_politics, international_law, sociology, us_foreign_policy).

Table 6. Chinchilla sets a new SOTA on MMLU, outperforming models 2.5× its size.

Other Benchmarks

• LAMBADA: 77.4% (Gopher 74.5%, MT-NLG 530B 76.6%)
• RACE-h: 82.3% (Gopher 71.6%, +10.7%)
• BIG-bench: 65.1% average (Gopher 54.4%, +10.7%)
• Natural Questions (64-shot): 35.5% (Gopher 28.2%, GPT-3 29.9%)
• TruthfulQA (0-shot): 43.6% (Gopher 29.5%, +14.1%)

Inference and Fine-tuning Cost

Because Chinchilla is 4× smaller than Gopher (70B vs 280B), it requires 4× less memory and FLOPs for inference and fine-tuning, making it significantly more practical for downstream applications.

Critical Analysis & Conclusion

Summary of Contributions

• New scaling law: Model size and training data should be scaled equally with compute budget (a≈0.5, b≈0.5), contradicting the previous Kaplan et al. law.
• Chinchilla validation: Empirical verification at 10^24 FLOPs scale, showing that a smaller model trained on 4× more data outperforms much larger models.
• Practical impact: Reinforces the importance of high-quality dataset scaling and provides a clear recipe for compute-optimal training.

Limitations

1. Limited large-scale verification: Only two comparable runs at scale (Chinchilla and Gopher); intermediate scales are missing.
2. Power-law assumption: The observed curvature in the FLOP-loss frontier (Appendix E) suggests the optimal model size may be even smaller for very large compute budgets.
3. Single epoch assumption: All models are trained for less than one epoch; the effect of multiple epochs (e.g., on datasets like Wikipedia) is not explored.
4. Dataset quality: The analysis assumes data quality is uniform; in practice, scaling to trillions of tokens requires careful filtering and deduplication.

Future Outlook

Chinchilla's findings have already influenced the design of subsequent LLMs (e.g., Llama 1 and 2, which used more data and smaller models). The key implication for the field is that data is not a free resource — we must invest in curating larger, cleaner, and more diverse datasets. Additionally, the scaling law is likely universal across modalities (vision, audio, multimodal), and similar compute-optimal analyses should be performed for non-text domains.

The paper also raises important ethical considerations: larger datasets scraped from the web contain more toxic and biased content, and careful mitigation strategies are needed. Chinchilla, despite being more capable, still exhibits bias and toxicity, though interestingly at a slightly lower rate than Gopher.

In summary, Chinchilla is a must-read for anyone training large models — it provides a simple, empirically grounded formula for allocating compute between model size and data, and demonstrates that sometimes bigger is not better.