[arXiv 2026] WRAP++: Amplifying Pretraining via Cross-Document Knowledge Discovery

Executive Summary

TL;DR: WRAP++ breaks the "single-document bottleneck" of synthetic pretraining. By following the "breadcrumbs" of web hyperlinks, it identifies related entities and forces an LLM generator to synthesize complex, multi-hop QA pairs. This approach amplifies a modest 8.4B token Wikipedia corpus into a massive 80B token relational dataset, leading to a +9.8% boost in factual accuracy (SimpleQA) and a much healthier scaling trajectory than previous SOTA methods.

Context: In the landscape of LLM pretraining, we are moving from "more data" to "better data." WRAP++ transforms the role of synthetic data from simple cleaning/rephrasing to active knowledge discovery, positioning itself as a critical technique for "upcycling" existing high-quality corpora.

The Problem: The Single-Document Bottleneck

Prior works like WRAP and Phi-4 have proven that rewriting messy web text into clean QA formats accelerates pretraining. However, they share a fundamental flaw: Isolation.

1. Limited Context: If a fact is split across two pages, a single-document rewriter can never "connect the dots."
2. Scale Saturation: You can only rewrite a Wikipedia page so many ways before you run out of new info. This leads to early plateaus in model performance.

Methodology: Discovering the "Relational Motifs"

WRAP++ treats the web as a Knowledge Graph. Instead of picking random pages, it searches for two specific topological patterns that signal a deep relationship:

• Dual-links ($A\leftrightarrow B$): Entities that reference each other (e.g., a Director and their hit Movie).
• Co-mentions ($A\to E\leftarrow B$): Entities that both point to a shared hub, combined with a direct link (e.g., two rival scientists mentioned in a discovery article).

Synthesis Constraints

To ensure the data isn't just "fluff," the authors enforce three strict rules during QA generation:

1. Cross-Document Dependency: The question cannot be answered using only one of the documents.
2. Explicit Factual Chaining: The model must "think out loud," showing the chain of logic from Doc A to Doc B.
3. Omniscient Internalization: The model must speak as a source of truth, not a student summarizing a passage (avoiding "According to the text...").

Key Results: Sustained Scaling

The most striking result is the Scaling Curve. As seen below, standard single-document methods (WRAP) stop providing benefits after ~17B tokens. WRAP++ continues to improve the model's factuality even at 80B tokens.

In the SimpleQA benchmark, which measures short-form factuality:

• OLMo-3-7B: Improved from 34.7 to 49.1 (+14.4 pts over base).
• OLMo-3-32B: Improved from 42.3 to 53.9 (+11.6 pts over base).

Ablation Insight: Random vs. Topological

Is the graph structure actually necessary? The authors tried pairing random documents, and the performance dropped significantly. Random pairing leads to "hallucinated relationships," while topological discovery ensures the synthesis model has a "semantic anchor" to work with.

Deep Insight & Conclusion

Why it Works

WRAP++ succeeds because it provides the model with diverse retrieval paths. By creating QA that bridges Hans Zimmer and Christopher Nolan, the model isn't just memorizing names; it is learning the structure of the film industry. This "relational supervision" allows the model to internalize multi-hop knowledge that it would otherwise have to infer implicitly from massive, noisy datasets.

Limitations & Future Work

The current study is focused on Wikipedia—a very "clean" graph. Extending this to the Wild Web (Common Crawl) will be the next frontier, as hyperlinks there are often noisy (ads, navigation). However, WRAP++ provides a clear blueprint for how we can move past the data wall by using the combinatorial power of entity relationships.

Final Takeaway: For the next generation of LLMs, the "Web Topology" might be as important as the "Web Content."