[ICLR 2026] Scorio: Navigating the Statistical Minefield of Reasoning LLM Leaderboards

TL;DR

As LLMs move toward "test-time scaling" (sampling dozens of attempts to find a correct answer), the way we rank them on leaderboards is breaking. This paper formalizes dense benchmark ranking, introduces the Scorio library, and reveals that while most methods agree at high budgets, your choice of "priors" and "scoring rules" can drastically shift results when you only have a few samples per question.

Background: The Shift from "What" to "How Stable"

In the traditional "Pass@1" era, ranking LLMs was simple: compute the accuracy and sort. However, reasoning models like DeepSeek-R1 or OpenAI’s o1-series change the game. We now care about Test-Time Scaling, where we might sample 80 times ($N=80$) to solve a math problem.

The authors argue that this turns evaluation into a repeated-sampling problem. The core challenge is no longer just "can it solve this?" but "how stably does our ranking method converge as we increase the inference budget?"

The Problem: The Instability of "The Top 1"

Existing leaderboards (like Chatbot Arena) often use Bradley-Terry models on sparse data. But on dense benchmarks (where every model tries every question), we have a 3D response tensor: Models $imes$ Questions $imes$ Trials.

When the budget is low (e.g., $N=1$), the ranking is noisy. If you use a single greedy decode as a "prior" to stabilize the rank, you might accidentally bias your leaderboard against models that perform better in stochastic "thinking" modes than in deterministic greedy modes.

Methodology: The Scorio Framework

The researchers implemented 72 different methods in a library called Scorio, categorized into several families:

1. Pointwise: Simple averages and difficulty-weighted scores.
2. Pairwise: Bradley-Terry, Elo, and Glicko (standard in sports).
3. IRT (Item Response Theory): Modeling latent model "ability" vs. question "difficulty."
4. Graph/Spectral: PageRank and Rank Centrality, treating model wins as directed edges.

The Model Architecture (Evaluation Flow)

The framework operates by transforming the raw response tensor into different representations (Pointwise, Pairwise, or Setwise) before applying the ranking engine.

Figure 1: Comparison of method agreement (Kendall’s $a{u}_b$) on easy vs. hard benchmarks. Notice how on hard benchmarks (HMMT'25), methods diverge significantly.

Key Insight: The Greedy Prior Trade-off

One of the most practical findings involves BayesR0 @N. By using a single "Greedy Decode" as an empirical prior for the Bayesian model, the authors found they could reduce ranking variance by up to 52% at $N=1$.

The Catch: This is a "shrinkage" estimator. It pulls the ranking toward the greedy performance. If a model's "greedy" output is poor but its "sampled" reasoning is brilliant, this prior will unfairly penalize it. This Bias-Variance trade-off is critical for anyone designing a robust LLM leaderboard.

Experimental Results: SOTA Comparison

The authors tested 20 models (including DeepSeek-R1, Qwen3, and Phi-4) on Olympiad-level math (AIME, HMMT).

Figure 2: Accuracy distributions across 80 trials. The wide error bars highlight why a single sample is insufficient for stable ranking.

Critical Findings:

• Consensus: At $N=80$, most methods agree with a 0.95 correlation.
• The Winner in Low-Budget: BayesR0 @N is the most stable for easier benchmarks, but IRT (Rasch MML) shows superior self-consistency on harder tasks.
• Categorical Signals: Incorporating "completion tokens" or "verifier confidence" into the rank (Categorical Ranking) makes the leaderboard more stable (self-consistent) but can decouple the ranking from actual "correctness."

Deep Insight: Why Bradley-Terry isn't enough

The paper provides a fascinating mathematical proof (Appendix C.2) showing that Average Accuracy and Bradley-Terry can disagree even with infinite data. Bradley-Terry focuses on "decisive wins" (one model correct, one wrong), while Accuracy focuses on marginal success. This proves that there is no "objective" ranking—only different statistical lenses.

Takeaways & Conclusion

1. Use BayesU @N as the default: It is the most interpretable and assumes the least about your data structure.
2. Audit your priors: If using greedy decodes to stabilize your leaderboard, check for "Greedy-Sampling Alignment" first.
3. Library Release: Scorio is a "batteries-included" tool for any AI lab wanting to move beyond noisy leaderboards.

Limitations: The study is math-heavy. Applying these to "vibe-based" or creative writing tasks where "correctness" is fuzzy remains a future challenge.

Blog written by [Senior Academic Tech Editor]