[CVPR 2026] $OneMillion-Bench: Are AI Agents Truly Ready for Professional Labor?

Executive Summary

TL;DR: The transition from LLM assistants to "Agentic" workflows requires more than just high scores on MMLU; it requires professional reliability. **$OneMillion-Bench\ast \ast isagroundbreakingbenchmarkfeaturing400specializedtasksacrossFinance,Law,andScience.Byanchoringevaluationtoactuallaborcosts($1M+ in expert wages), it provides a "value-faithful" audit of whether AI can replace or augment human experts.

Positioning: This work moves beyond "Exam-style AI" to "Workflow AI." It is a diagnostic testbed for the reliability, economic utility, and domain-depth of frontier agents like Claude-Opus, GPT-5, and specialized Deep Research models.

Problem & Motivation: The "Exam-Task" Delusion

Most current benchmarks (GPQA, MMLU-Pro, HLE) treat AI like a student taking a test—single-turn, isolated, and "Google-proof." However, professional labor is context-heavy and multi-step. A lawyer doesn't just "know" the law; they must verify cross-border compliance clauses against specific, moving targets.

The authors argue that traditional benchmarks have reached saturation. We no longer need to know if an AI can pass a medical exam; we need to know if it can handle an actuary's valuation model under IFRS 17. The core "pain point" is the lack of a metric that translates model capability into an interpretable quantity of delivered work value.

Methodology: Coding Professionalism into Rubrics

The core innovation lies in the Expert Score and Economic Value metrics.

1. The Economic Anchor

Every task in $1M-Bench is assigned a dollar value based on: $V=T_{extExpertCost}imes{W}_{extHourlyWage}$ This allows researchers to plot a Pareto Frontier of inference cost vs. economic return.

2. Multi-Dimensional Rubrics

Unlike simple BLEU or Accuracy scores, the benchmark uses domain-specific rubrics including:

• Factual Information (FI): Tracking authoritative sources.
• Analytical Reasoning (AR): Causal attribution and logic.
• Instruction Following (IF): Adhering to professional constraints.
• Negative Rubrics: Penalizing "Bad Behaviors" like violating industry norms or hallucinating figures.

Figure 1: The Three-Stage Data Curation Pipeline including Peer Review and Adversarial Validation.

Experiments & Results: The Search Amplifier

The benchmark evaluated 35 systems, categorized into Vanilla models, Search Agents, and Deep Research Agents.

Key Insights:

1. Search is an Efficacy Amplifier: For strong models (Claude/GPT), Search tools led to dramatic score gains. However, for weaker models, Search introduced "noise," leading to performance regressions.
2. Generalists vs. Specialists: Surprisingly, general-purpose models with web search (Claude-Opus-4.6) often outperformed specialized "Deep Research" models. This suggests that "Rubric Compliance" is currently a bigger bottleneck than the depth of search.
3. The "Near-Miss" Phenomenon: Many models achieved a moderate Expert Score (~50%) but failed the Pass Rate (requires Expert Score ≥ 0.7). This means AI is "vaguely professional" but lacks the precision to complete a task to a standard a human supervisor would accept.

Figure 2: Performance on Global and CN subsets. Claude-Opus-4.6 leads the pack in economic value and expert utility.

Deep Insight: Failure Patterns

The paper’s case studies reveal where AI hits the "glass ceiling":

• The "Double-Edged" Search: In Finance, search results often pull models toward technical analysis (RSI/Moving Averages) when the rubric requires deep fundamental logic (BoJ policy rates), causing a drop in analytical reasoning scores.
• Normative Blindness: In Law, models struggle with "Imprecise Mapping"—failing to apply a specific 2024 Supreme People’s Court Guiding Case despite knowing the general law.
• Calculatory Fragility: Even the best models stumble on "cross-company comparison of inventory turnover days," showing that workflow fluency is easily derailed by basic arithmetic errors.

Conclusion & Future Work

$OneMillion-Bench is a wake-up call for the AI community. The "Pass Rate" across most models remains below 45%, showing that we are still far from autonomous "Human Expert" agents.

Takeaway for Practitioners: When deploying agents in vertical domains, the focus should shift from "more search" to "better rubric compliance" and "negative-behavior masking." The next frontier is not just more tokens, but value-faithful decision making that survives the scrutiny of a senior actuary or lawyer.

Disclaimer: This analysis is based on the arXiv paper "$OneMillion-Bench: How Far are Language Agents from Human Experts?" produced by Humanlaya, BIGAI, and partners.