[Stanford Research] Beyond More Tokens: Scaling Synthetic "Megadocs" for 1.8x Pre-training Efficiency

TL;DR

In a world where we are running out of internet text to train AI, Stanford researchers have found a way to make every real token go nearly twice as far. By transforming single web documents into structured "Megadocs"—long sequences of synthetic rephrases and "latent thoughts"—they achieved up to 1.80x data efficiency. Unlike simple data augmentation, this approach scales better with compute and significantly boosts performance on both standard benchmarks and long-context tasks.

The Problem: The Synthetic Plateau

Pre-training is no longer just about compute; it is about the data wall. We are consuming web text faster than it is being produced. While using synthetic data (like rephrasing the web) has helped, it usually hits a "plateau."

Current methods treat every synthetic rephrase as an independent, shuffled document. This ignores a vital piece of information: structure. If you have 10 versions of the same Wikipedia article, the relationship between those versions is just as important as the text itself.

The Innovation: Constructing the Megadoc

The researchers propose moving away from "shuffling" and toward "stitching." They introduce two primary architectures for synthetic data:

1. Stitched Rephrasing: Instead of treating 32 rephrases as 32 separate docs, they stitch them together into one massive document. Their "Aha!" moment? Putting the real document last performs best, likely because it forces the model to "recover" the high-detail original text from the lower-detail synthetic rephrases.
2. Latent Thoughts: Inspired by "Chain of Thought," they split a real document into pieces and ask a generator to write "latent thoughts" (background knowledge and reasoning) to bridge the gaps. This effectively "stretches" the document from within.

Figure 1: While simple rephrasing plateaus (orange), Megadocs (blue/gray) continue to scale as more synthetic data is added.

Why It Works: The "Infinite Training" Hack

The core technical insight isn't just that the data is "better." It’s that Megadocs change the hyperparameter landscape.

Typically, if you train on the same data for too many epochs, the model overfits. However, Megadocs allow for:

• Higher Mixing Fractions: You can train on up to 90% synthetic data without losing the original distribution.
• More Epochs: Megadocs allowed the researchers to train for 5x more steps on the same amount of real data before overfitting.

This suggests that Megadocs act as a sophisticated form of regularization, providing a "harder" and more diverse learning task that prevents the model from simply memorizing the real tokens.

Experimental Results: Breaking the 1.5x Barrier

Moving from 300M to 1.5B parameter models, the researchers confirmed that their "Megadoc" win actually increases with model size.

Figure 2: Performance across i.i.d loss, long-context arXiv loss, and downstream benchmarks. Latent Thoughts (gray) consistently leads.

• Long-context Superiority: On arXiv CS papers, Latent Thoughts improved loss by 0.19—a massive margin in pre-training terms.
• Ensemble Composition: Unlike "Self-Distillation," which fails to improve when you ensemble multiple models, Megadoc training composes with ensembling. This means it provides a "unique" signal that ensembling alone cannot capture.

Critical Analysis & Future Outlook

While the results are impressive, the study relies on a "Teacher" model (Llama 3.1 8B) that is stronger than the "Student" (300M - 1.5B). Critics might call this "distillation in disguise."

However, the authors provide a compelling rebuttal: synthetic augmentation in CV (like blurring or flipping images) helps at every scale. They argue that once we reach a certain threshold of compute, these rephrasing and "thinking" capabilities become a commodity that any model can use to improve itself.

The Takeaway: If you are data-constrained, don't just generate more text. Generate longer, structured contexts that force your model to understand the logical flow between variations of the same idea. The future of pre-training isn't just "more tokens"—it's "better structures."

Paper: Data-efficient pre-training by scaling synthetic megadocs (Kim et al., 2025)