BENCHPRESS: Why You Don’t Need to Run Every LLM Benchmark

TL;DR

Evaluating a modern AI model is like putting a student through 100 different exams just to find out they are "good at everything." This paper introduces BENCHPRESS, a tool that proves the LLM "scorecard" is mostly redundant. By running just five specific benchmarks, BENCHPRESS can predict a model's performance on over 100 others with a median error of less than 4 points.

Background: The Evaluation Explosion

We are currently in a "Benchmark Arms Race." Models like Qwen 3.5 or Claude 3.7 report scores on 40+ benchmarks to prove their worth. For researchers, this represents thousands of dollars and days of compute time. The authors of "You Don't Need to Run Every Eval" asked a simple, provocative question: Is model performance actually high-dimensional, or are we just measuring the same two things over and over again?

The "Rank-2" Insight: Model Performance is Simpler Than It Looks

The most striking discovery in this paper is the Rank-2 Geometry of the score matrix. After compiling a matrix of 84 frontier models and 133 benchmarks, the authors used Singular Value Decomposition (SVD) to analyze the underlying patterns.

They found that two factors explain over 90% of the variance between models. In physical terms, most of what makes a model "good" boils down to two axes:

1. General Intelligence: The rising tide that lifts all boats.
2. Reasoning/Math Specialty: The extra "kick" found in chain-of-thought and reasoning-heavy models.

Figure 1: BENCHPRESS predicts unseen benchmark scores from a handful of revealed ones.

Methodology: How BENCHPRESS Predicts the Unknown

To turn this insight into a tool, the authors developed a matrix completion method called Logit Bias ALS.

1. Logit Space: Since benchmark scores (0-100%) are non-linear (it’s harder to go from 98% to 99% than 50% to 51%), they transform the data into "Logit Space" to linearize it.
2. Bias Decomposition: The model accounts for the fact that some benchmarks are naturally harder (benchmarking offset) and some models are naturally smarter (model offset).
3. Low-Rank Correction: It then uses the rank-2 residual to fine-tune the interactions between a specific model and a specific type of task.

Results: The "Five-Benchmark" Scorecard

One of the most practical outputs of this research is identifying the "Probe Set." If you have a brand-new model and a limited budget, which tests should you run?

The authors found a "Golden Five" set: {GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1}. Running just these five allows BENCHPRESS to fill in the rest of a 133-benchmark scorecard to within 3.93 points of accuracy.

Figure 5: Error drops sharply as specific "probe" benchmarks are revealed, allowing for high-accuracy reconstruction.

Critical Insight: When Should You Trust the Prediction?

The authors didn't just build a predictor; they built a Confidence Layer. Through a hybrid reliability estimator, BENCHPRESS can tell you when it's guessing.

• High Trust: If the model has a "neighbor" in the matrix (e.g., GPT-4.5 vs GPT-4) and the benchmark has lots of previous data.
• Low Trust: If the model is a completely new architecture or the benchmark is a "lone wolf" with low correlation to others.

Conclusion: A New Era of "Inferred Evaluation"

This paper is a wake-up call for the AI industry. It suggests that while benchmarks are necessary for science, they are often redundant for selection and tracking training progress.

The Takeaway: Instead of wasting thousands of dollars on 100 benchmarks for every checkpoint, engineers can use BENCHPRESS to run a "Lite" version of their evaluation suite and infer the rest with statistical confidence.

Limitations: As the authors note, this rank-2 geometry is a snapshot of today's models. If a future model discovers a "third dimension" of intelligence (e.g., true social emotional intelligence or novel physical reasoning), the matrix will need to be re-calculated.