Visual Pretraining: Breaking the Text-Only Ceiling of Language Intelligence
Scalable Visual Pretraining for Language Intelligence
The paper introduces Visual Pretraining (VP), a scalable framework that trains foundation models directly on raw document images (PDFs) using a next-visual-latent prediction objective. By bypassing lossy text extraction, VP achieves new SOTA results in scientific reasoning across backbones like Qwen 3.5 and Llama 3.2.
TL;DR
Researchers have long assumed that "text is all you need" for language intelligence. This paper shatters that assumption by showing that Visual Pretraining (VP)—training models on raw document images instead of parsed text—consistently produces smarter models. By training on the same scientific corpora, VP outperforms text-based training while consuming 75% fewer tokens.
The "Textualization" Tax: What LLMs Lose
When we train an LLM on a scientific paper, we usually "textualize" it: we strip away the font sizes, the spatial layout of equations, and the geometric relationships in diagrams. This process is inherently lossy. A complex Maxwell equation or a biological flowchart contains spatial cues that help humans reason; converting these into a flat string of characters forces the model to reconstruct 3D logic from 1D data.
The authors argue that this "projection" of a rich visual structure into plain text creates an intelligence bottleneck. Their solution? Stop extracting text and start training the model to predict the next visual patch of the actual document page.
Methodology: Autoregressive Latent Prediction
The core of VP is surprisingly elegant. Instead of predicting the next word, the model predicts the next visual latent.
1. Sparse Document Representation
Not every pixel on a page is important. Most of a PDF is white space. VP uses a Foreground-token filtering mechanism that identifies patches with high variance (content) and ignores the rest. These patches are then ordered via a raster-scan (left-to-right, top-to-bottom) and projected into the LLM's hidden space.
2. Next-Visual-Latent Objective
The model is tasked with predicting the latent representation of the next patch. This is done using a contrastive InfoNCE loss, which encourages the model to distinguish the correct next patch from other patches in the batch.
Figure 1: The dual pathway approach. While Text Pretraining (TP) distorts layouts, Visual Pretraining (VP) preserves the geometric integrity of the source document.
Efficiency and Results: Doing More with Less
The most striking result of this paper is efficiency. On a matched corpus of scientific PDFs, the text version resulted in 80 billion tokens, whereas the visual version (retaining only foreground patches) resulted in only 20 billion tokens.
- Direct Gains: VP led to a +3.22 point increase on GPQA (Graduate-level QA) and an +8.94 point jump on AIME (Math Competition) for Llama 3.2 Vision compared to its base.
- Scaling Laws: VP's advantage over text training actually increases as you add more data and compute.
- Structure Density: The performance gap is widest on documents with high "visual-structure density"—those filled with complex tables and diagrams.
Figure 2: VP's advantage is most pronounced in structure-heavy pages, proving that it successfully internalizes visual reasoning cues.
Emergent Multimodality
Even though VP is performed without any explicit "image-text" pairs (labels), the model naturally learns to align these two modalities. By testing on multimodal benchmarks like MathVista and ChartQA, the researchers found that VP-trained models significantly improved their visual perception capabilities simply by learning to predict document patches. The modality gap—the distance between how a model sees "text" vs "images"—collapsed significantly.
Critical Insight: Beyond Pixels
One might ask: why not just use a generative decoder to reconstruct the pixels (like an MAE)? The authors tested this (VP with Decoder) and found it was 30-40% slower with marginal benefits. The takeaway is profound: predicting the structure (latents) is more important than reconstructing the pixels.
Conclusion & Future Outlook
This work signals a shift in the foundation model paradigm. We are moving away from the "OCR-then-train" era toward a "native-visual" era. If a model can learn human-level reasoning better by "looking" at a page than by "reading" its transcript, the future of data scaling might not lie in more text, but in more high-quality visual streams.
Limitations: Currently, VP is most effective for high-density scientific data. Whether this translates to natural images (cats, dogs, landscapes) remains an open research question.