[Pretraining Science] daVinci-LLM: Deciphering the Data Darwinism that Powers 3B Models to 7B Performance

Executive Summary

TL;DR: The daVinci-LLM project bridges the gap between massive-scale commercial "black boxes" and small-scale academic transparency. By training a 3B model on 8 trillion tokens and documenting over 200 ablation studies, the authors demonstrate that data processing depth (quality over quantity) and adaptive curricula allow a 3B model to match the performance of the prestigious 7B OLMo-3.

Background Positioning: This is a foundational "Science of Pretraining" paper. It doesn't just chase SOTA; it charts the map of how to get there, providing the community with the "Data Darwinism" framework to navigate the transition from raw text to structured reasoning.

The Problem: The Pretraining Paradox

Current LLM development suffers from a transparency crisis. We have "Open Weights" (Llama, Qwen) but rarely "Open Recipes." Most organizations treat the specific data mixtures and training rhythms as trade secrets. This prevents the research community from understanding why certain models succeed while others fail. Specifically, we lack a systematic way to measure the impact of data quality vs. data volume.

Methodology: Data Darwinism & Adaptive Curricula

The core innovation is the Data Darwinism Framework, which classifies data processing into 10 levels (L0–L9):

• L3 (Model-based Filtering): Using classifiers to pick the best documents.
• L4 (Generative Refinement): Using larger models to "clean" and "reorganize" scientific text into logical, information-dense formats.
• L5 (Cognitive Completion): Actively synthesizing reasoning chains (QA pairs) to bridge gaps in expert-level technical documents.

Two-Stage Training Recipe

1. Stage 1 (General Foundation - 6T tokens): Focused on linguistic fluency using diverse web corpora.
2. Stage 2 (Reasoning Enhancement - 2T tokens): A pivot toward structured logic, using a 70% concentration of QA and refined scientific data.

Figure: The mapping of data sources onto the Data Darwinism taxonomy across stages.

Key Insights from 200+ Ablations

The authors' extensive testing revealed several "laws" of pretraining:

• The Saturation Law: General knowledge benchmarks plateau early (around 1T tokens), while reasoning (code/science) continues to grow. This justifies shifting the data mixture mid-stream.
• Depth Over Volume: L4 generative refinement on math data yielded a +7.00 jump on MATH, proving that transforming content is more effective than simply adding more raw data.
• The QA Advantage: Introducing structured QA pairs in the late stages of pretraining prevents performance plateaus that occur when only using unstructured text.

Experimental Results: Punching Above Its Weight

daVinci-LLM-3B doesn't just compete with its 3B peers; it destroys them in reasoning tasks and matches 7B models.

Figure: daVinci-LLM-3B vs. Baselines. Note the massive lead over Llama-3.2-3B and its parity with OLMo-3-7B.

Critical Analysis & Conclusion

Takeaways: The success of daVinci-LLM suggests that the future of base models lies in curation-centric scaling. By treating data processing as an evolutionary process (Darwinism), we can achieve superior reasoning in smaller, more efficient architectures.

Limitations: The authors note that extreme specialization in QA data during Stage 2-1 can lead to "domain collapse" if not balanced carefully. High-intensity supervision requires a stable representational base first.

Future Outlook: The release of intermediate checkpoints (every 5k steps) and the full data pipeline provides a rare "fossil record" of LLM development, allowing the community to study the emergence of intelligence in real-time.