[ACL 2025 Submission] SCAN: Beyond the Leaderboard Obsession—A Diagnostic Map for LLM Capabilities

TL;DR

The LLM community is currently trapped in a "ranking race," where a single Elo score determines a model's worth. SCAN (Structured Capability Assessment and Navigation) breaks this paradigm by providing a high-resolution "medical check-up" for LLMs. By combining an automated hierarchical taxonomy, a realistic query synthesizer called RealMix, and a revolutionary PC2 (Pre-Comparison-derived Criteria) judging mechanism, SCAN reveals the "spiky" reality of model performance that aggregate scores hide.

The "Lighthouse" Problem: Why Rankings Mislead

Evaluation has hit a bottleneck. Human-led leaderboards like Chatbot Arena are too slow for rapid development cycles, while automatic benchmarks like MT-Bench are often too coarse. A model might rank #1 overall but fail miserably at Bioengineering-specific queries or Java threading.

The authors argue that we need to move from Ranking to Understanding. Current methodologies either sacrifice accuracy (Pointwise) or scalability (Pairwise). SCAN enters this gap as a scalable, fine-grained diagnostic framework.

Methodology: The SCAN Architecture

SCAN is built on three pillars that transform raw queries into actionable insights:

1. TaxBuilder: Recursive Taxonomy Construction

Instead of a flat list of tasks, TaxBuilder uses a recursive node insertion mechanism. By behaving like a self-balancing tree in data structures, it allows an LLM to decide whether a new capability (e.g., "React Hooks") is an expansion of an existing node, a new sibling, or a child. This avoids the long-context window issues of feeding an entire taxonomy into a prompt.

2. RealMix: Synthesis without Contamination

To avoid using public test sets that models might have "memorized" during training, SCAN uses RealMix. It takes high-quality real-user query content and "mixes" it with specific tags from the taxonomy. This ensures the evaluation set is both diverse (covering the long tail) and clean.

3. PC2 Judging: Borrowing Logic from Pairwise Evaluation

The most technical innovation is PC2.

• The Logic: In pairwise evaluation, the judge sees two answers and finds the "difference points." In pointwise, the judge is "blind" to alternatives.
• The Hack: PC2 generates several auxiliary responses first. The judge compares these to derive weighted criteria (e.g., "Code Efficiency: 40%, Syntax: 60%"). Then, it scores the target model against these specific, dynamic metrics.

Experimental Revelations: The GPT-OSS Case Study

The authors applied SCAN to 21 models, including the newly released GPT-OSS family. The results were eye-opening:

• The Generalist's Trap: GPT-OSS-120B is the "overall" king, but SCAN's Failure Mode Explorer flagged a massive variance in its knowledge domain. It excels in Computer Science but ranks #11 in Bioengineering.
• The Small Model Secret: GPT-OSS-20B (a smaller model) actually outperformed its 120B counterpart in C and C# coding. Without SCAN's fine-grained view, we would assume the 120B model is superior across the board.

Deep Insight: Is Pointwise Back?

For a long time, the consensus was that pointwise scoring was "broken" because LLMs are biased Toward high numbers. SCAN's PC2 method proves that pointwise evaluation isn't fundamentally broken—it was just context-poor. By providing the judge with "comparison-derived criteria" before the score, SCAN achieves the reliability of Chatbot Arena with the speed of a script.

Conclusion & Limitations

SCAN is a major step toward "Precision AI Engineering." However, the authors admit that it currently lacks Multimodal support (visual/audio) and doesn't explicitly measure Safety/Factuality dimensions yet.

The project, available at https://anonymous-scan.github.io/SCAN/, provides the community with a "Failure Mode Explorer" that every LLM developer should probably have in their CI/CD pipeline.