[CVPR 2026] Nexus: Achieving Better Downstream Generalization with the Same Pretraining Loss via Common Minima

TL;DR

In the era of Large Language Models (LLMs), the industry has long relied on Pretraining Loss as the ultimate North Star. However, a groundbreaking paper from Tsinghua and ByteDance Seed reveals a hidden dimension: Geometric Closeness. By introducing the Nexus optimizer, the researchers demonstrate that models can achieve dramatically better reasoning performance (e.g., +15% on GSM8k) despite having the exact same pretraining loss as standard AdamW. The secret lies in steering the model toward the "Intersection of Minima" rather than just the "Sum of Minima."

The Geometric Intuition: Sum vs. Intersection

The central question of the paper is profound yet simple: When we minimize a summed loss $L_{train}=\frac{1}{K}\sum L_k$, where do we actually land?

The authors distinguish between two types of convergence:

1. Sum of Minima: The model finds a point where the total loss is low, but it is far from the optimal point of any single domain (e.g., it's "okay" at math and "okay" at code, but doesn't hit the sweet spot for either).
2. Intersection of Minima (Common Minima): The model converges to a point that is simultaneously a minimizer for all individual data sources.

The researchers hypothesize—and mathematically prove for quadratic surfaces—that Closeness (the distance to these task-specific minima) is a second-order property that dictates downstream generalization.

Methodology: Maximizing Consensus via Nexus

Directly calculating the distance to every task's potential minimum is computationally impossible. Nexus bypasses this by using Gradient Similarity. If two tasks have highly aligned gradients throughout training, they likely share a common minimum.

The Nexus Mechanism

Nexus operates through a dual-loop mechanism:

• Inner Loop: It takes several steps of Normalized SGD (NSGD). This sequence effectively explores the local curvature (the Hessian).
• Outer Loop: It computes the displacement between the start and end of the inner loop. This displacement acts as a "pseudo-gradient" that contains a regularizer for gradient similarity.

Crucially, for engineering efficiency, the authors adapted this into a gradient accumulation workflow (Algorithm 3 in the paper). By keeping an auxiliary inner_model and calculating the "drift" from the main model, they achieve these second-order benefits with nearly zero additional computational cost.

Experimental Results: Breaking the Loss-Performance Correlation

The most striking result of Nexus is its ability to break the traditional scaling law where "lower loss = better performance."

Key Findings:

• Reasoning Surge: On a 3B model, while the pretraining loss remained unchanged (1.606 vs 1.602), the accuracy on GSM8k (Math) jumped from 44% to 59%.
• Scaling Favorability: The gap between Nexus and AdamW grows as models get larger. At 130M, the gain was +0.8%; at 3B, the gain was +3.2%.
• Data Mixture Robustness: Nexus acts as an "implicit balancer." If math data is scarce (e.g., 10%), Nexus works harder to "mine" those rare signals, significantly outperforming AdamW in sample-sparse domains.

Critical Insights: Why Does This Matter?

For years, the AI community has focused on Flatness (finding wide minima). This paper introduces Closeness as a complementary and equally vital factor.

Nexus proves that the path taken by the optimizer matters as much as the final loss value. By forcing tasks to "agree" (gradient consensus), the model develops representations that are fundamentally more robust and capable of handling complex reasoning, which often requires synthesizing knowledge from multiple domains.

Limitations & Future Work

While Nexus is a "drop-in" upgrade for AdamW, it currently faces compatibility issues with Muon, another high-performance optimizer. The authors also note that while they reduce geometric distance, they have not yet reached the "perfect intersection." Future research into even more aggressive consensus-seeking optimizers could push LLM capabilities even further without requiring more data or compute.

Conclusion

Nexus represents a shift in LLM pretraining from "compute-bound" thinking to "geometric-bound" refinement. It suggests that our current models are undertrained not in terms of tokens, but in terms of inter-task synergy. For practitioners, adding a Nexus-style regularizer might be the most cost-effective way to boost reasoning performance in the next generation of models.

Note: This blog is based on the paper "Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima" (April 2026).