Which pretraining corpus regions support social reasoning versus STEM reasoning in LLMs?

Social reasoning sits apart from the other three benchmarks

The paper asks where in a pretraining corpus different capabilities find their support. It answers by computing gradient-based training-data attribution for four benchmarks in OLMo3-7B, then aggregating document-level influence into 576 topic–format bins defined by the WebOrganizer taxonomy (24 topics crossed with 24 formats). The four benchmarks form a 2×2 design: domain (social versus STEM) crossed with capability type (reasoning versus knowledge). SocialIQA and ARC-Challenge are the reasoning cells; MMLU Social Sciences and MMLU STEM are the knowledge cells.

The headline result is a separation, not a single winning region. SocialIQA's full-grid attribution profile correlates with the three comparison benchmarks at r = 0.06, 0.21, and 0.09, while those three correlate with each other at r = 0.76–0.86. An outlier statistic that compares each benchmark's mean correlation to the others is positive only for SocialIQA, at +0.68 with a 95% bin-bootstrap interval of [+0.34, +0.96]. The same ranking holds when the analysis is repeated on format marginals and topic marginals, so the finding does not depend on one slice of the taxonomy.

That contrast is sharper at the reasoning level than at the knowledge level. The paper's own framing is that social and STEM reasoning draw on qualitatively distinct corpus regions, while the knowledge pair separates less cleanly. Readers should keep those two claims apart: the strong evidence is about SocialIQA versus the rest, not about a clean social-versus-STEM split across all four benchmarks.

Document-level influence is too noisy to read directly

Training-data attribution estimates how strongly each training document influences a model's predictions on a benchmark. The problem is granularity: individual document scores are noisy, so a ranked list of documents does not tell you which kind of corpus material supports a capability. The paper's move is to aggregate those scores into a fixed taxonomy, turning noisy per-document estimates into corpus units a reader can compare or perturb.

The scale of that aggregation matters for interpreting the numbers. Attribution runs over a stratified working set of 5,678,621 documents (10.5B tokens, 10,000 per bin) drawn from the de-duplicated Dolma3 mix of 1,263,587,168 unique documents. Each document receives a topic and a format from WebOrganizer, and the attribution unit becomes the resulting topic–format bin, such as Social Life × Q&A Forum or Literature × Customer Support.

Two implementation choices shape what the scores mean. Gradients are computed with OLMo3-7B Base for documents and OLMo3-7B Instruct for benchmark queries, and the supervised-instruction-tuning second-order contribution is treated as identity so that attribution is tied to pretraining data. Influence is also z-standardized within each benchmark, which is why the paper compares the positions of profiles rather than their magnitudes.

SocialIQA draws on dialogue-rich and expository regions

SocialIQA's positive influence is dispersed across many topic–format cells rather than concentrated in a few. Its top format is Customer Support (+1.07), which does not appear in any other benchmark's top three, and its high-influence bins are described as dialogue-rich. The paper reads this as interpersonal formats carrying social interaction, in contrast to the documentation-like formats that the other benchmarks rely on.

The comparison benchmarks show a different shape. Documentation is the strongest positive format for MMLU Social Sciences (z = +2.40), MMLU STEM (+2.29), and ARC-Challenge (+1.97), and remains mildly positive for SocialIQA (+0.44). The paper interprets documentation-like formats as carrying more direct factual content. News Article is a consistent negative across all four benchmarks, ranging from −0.62 to −1.02.

These are associations in attribution scores, not causal claims about individual documents. The paper is explicit that attribution estimates influence and does not prove causal necessity, which is why it adds an intervention step rather than stopping at the heatmaps.

Unlearning validates SocialIQA, not all four benchmarks

To test whether high-attribution regions are functionally necessary, the paper forgets documents from high-attribution topics using targeted machine unlearning and measures selectivity: damage on the intended benchmark relative to mean collateral damage on the other three. Higher selectivity means the intervention hurt the aligned benchmark more than the others.

The results are mixed and the paper reports them that way. SocialIQA is favorable under both forget-set constructions: a bin-level pilot gives 1.43× and a faithful per-document pilot gives 2.17 ± 0.61×. MMLU STEM is favorable only under the faithful per-document construction (1.45 ± 0.10×) and loses under the bin-level pilot (0.62×). MMLU Social Sciences loses under both (0.77× and 0.15 ± 0.02×). ARC-Challenge shows no selectivity advantage under either construction (0.82× and 0.87 ± 0.24×), and because its on-target damage sits at the noise floor, the paper treats it as an uninterpretable non-result rather than a failure.

A separate 2×2 recipe comparison isolates the two ingredients of bin-targeted unlearning: ranking documents by influence and restricting the forget set to one topic. Under the corrected evaluation, influence ranking helps on 1 of the 3 scorable benchmarks and topic filtering on 0 at the strict blind top-1 topic. The paper therefore describes its robustness checks as a mixed audit, not a uniform four-benchmark win.

The blind top topic is a weak target for both MMLU benchmarks

Part of the mixed unlearning result comes from which topic gets selected. For both MMLU Social Sciences and MMLU STEM, the blind top-1 attribution topic is Industrial, and the paper notes that this is a weak unlearning target for both. That is why their topic effects fall below 1.0 even though their attribution profiles differ from SocialIQA's.

The gap between deployable and best-case selection is large. A best-in-hindsight upper bound, which picks the best topic after seeing the outcome, reaches 5.11 ± 2.61× for SocialIQA, 1.88 ± 0.68× for MMLU Social Sciences, 1.45 ± 0.10× for MMLU STEM, and 4.85 ± 0.63× for ARC-Challenge. The paper labels this an upper bound, not a deployable selection rule.

This distinction is the practical core of the unlearning section. Attribution can point at regions that matter, but the current top-1 selection rule does not reliably convert that signal into a selective intervention for every benchmark. SocialIQA is the case where the pipeline works end to end.

The map describes a working set, not the whole corpus

The provenance claims are anchored to the de-duplicated Dolma3 population, but the empirical analysis runs on a stratified working set drawn from its WebOrganizer-binned portion. Results characterize benchmark-relevant structure within that working set and should not be read as exhaustively characterizing every document in the full corpus.

Weighting also changes the question. A post-hoc token-mass weighting check preserves the sign of each bin's influence but changes which bins dominate population-level totals, because high-volume bins can outweigh lower-volume high-z bins. SocialIQA's strongest balanced bin remains Literature × Customer Support, while the largest token-weighted positive contributions come from higher-mass technical and tutorial bins. The paper reports its main results as balanced taxonomy-level contrasts and treats corpus-weighted summaries as a separate population-scope diagnostic.

The bins themselves are classifier-defined compressions rather than exact semantic partitions, and the study covers one model family. The paper states that whether the high-level attribution patterns hold across other open-data models trained on web-scale corpora remains open, and that testing this is a separate follow-up rather than an incremental experiment.

Read SocialIQA as the validated case, not the template

The paper's most defensible claim is narrower than a general theory of where social versus STEM reasoning lives. Within the OLMo3-7B and Dolma3 setting, SocialIQA draws positive attribution from a broader, more evenly distributed set of topic–format regions than the comparison tasks, with influence from both interactional and expository writing, and targeted unlearning validates that result most clearly.

For readers comparing benchmarks or planning corpus interventions, the useful split is between the four targets. SocialIQA is favorable under both forget-set constructions; MMLU STEM is favorable only under faithful per-document forget sets; MMLU Social Sciences is an agreeing loss; ARC-Challenge is a noise-floor non-result. Treating all four as equally validated would overstate what the unlearning experiments show.

The paper's own next step follows from that split: repeat the pipeline on a larger model from the same family with an identical unlearning protocol, and test whether causal selectivity and category associations survive scale. Until then, the capability-provenance map is best used as a structured summary of where benchmark-relevant influence concentrates, with SocialIQA as the case where the map and the intervention agree.

Sources used in this answer