Databricks AI Research introduced OfficeQA Pro, a high-fidelity benchmark for evaluating AI agents on grounded reasoning over a 100-year corpus of U.S. Treasury Bulletins. The benchmark requires precise multi-document retrieval and analytical reasoning across 26 million numerical values, with the strongest agent (Claude Opus 4.6) achieving only 48.1% accuracy on raw PDFs.
TL;DR
Databricks AI Research has released OfficeQA Pro, a brutal new benchmark that proves even "GPT-5 class" models are hitting a ceiling in enterprise environments. By testing agents against a 100-year archive of U.S. Treasury Bulletins (89k pages, 26M values), they found that the strongest agents fail over 50% of the time. The secret to a 16% boost? It’s not better reasoning—it’s better parsing.
Background: The Gap Between "Smart" and "Reliable"
We’ve seen models solve International Math Olympiad problems, but can they tell you the absolute difference in U.S. national defense expenditures between 1940 and 1953, adjusted for inflation, using only the most recently revised Treasury figures?
Most benchmarks (like HLE or ARC-AGI-2) test abstract intelligence. OfficeQA Pro tests Grounded Reasoning: the ability to find a needle in a haystack of 89,000 pages and then perform precise "economic-grade" calculations on that needle.
The "Parsing Bottleneck": The Invisible Wall
One of the paper’s most provocative findings is that the document representation (how the AI "sees" the PDF) matters as much as the model's brain.
Figure 1: Even with the latest models, no agent surpasses 50% accuracy on the full corpus.
When agents were forced to parse raw PDFs using standard tools (like Tesseract), they spent hours installing libraries and ultimately failed due to "table topology failures"—shifted rows or missing headers in scanned documents from the 1940s.
The Insight: Structured Context
By using Databricks’ ai_parse_document, which converts complex tabular hierarchies into structured representations (HTML/Markdown), agents saw a 16.1% relative performance gain. This suggests that RAG (Retrieval-Augmented Generation) is a parsing problem first, and a retrieval problem second.
Figure 2: The diversity of layouts—from 1941 customs duties to 1980 public debt—creates massive Inductive Bias challenges for standard parsers.
Methodology: The Anatomy of an Enterprise Agent
The researchers didn't just test models; they tested System Design. Key experiments included:
- Table Serialization: HTML vs. Markdown. (HTML won slightly, likely due to its ability to represent nested hierarchies).
- Search Strategy: Comparing Vector Search, Contextual Embeddings, and File Search (grep/ls).
- Test-Time Scaling: Using plurality voting (N=4) to see if "thinking harder" or "voting" improved accuracy.
Results: The Reality Check
| Agent | Corpus Format | Correctness (%) | Latency (min) | | :--- | :--- | :--- | :--- | | Claude Opus 4.6 | Raw PDF | 48.12% | 31.2 | | Claude Opus 4.6 | Parsed (DBX) | 54.14% | 5.3 | | GPT-5.4 | Raw PDF | 36.09% | 13.1 | | GPT-5.4 | Parsed (DBX) | 56.39% | 3.6 |
Providing structured parsing didn't just make the agents better—it made them 4x to 9x faster.
Why Agents Still Fail: The "Revision Trap"
The most sophisticated failure mode identified is Temporal Revision Verification. In enterprise data, a "2011 Revenue" figure published in 2011 is often an estimate. The real number is in the 2012 or 2013 report. Agents often "prematurely converge" on the first plausible number they find.
Other failure modes include:
- Visual Logic: Agents are still blind to dense financial line plots (see Figure 10 in the paper).
- Calculation Drift: Using sample variance instead of population variance, or rounding too early in a multi-step OLS regression.
Critical Analysis & Conclusion
OfficeQA Pro is a wake-up call for the "Agentic Workflow" hype. It demonstrates that being a "frontier model" is not enough for enterprise-grade tasks.
The Takeaway: If you are building enterprise AI, stop obsessing solely over the LLM’s reasoning capabilities and start investing in your data ingestion pipeline. High-fidelity parsing and "revision-aware" retrieval are the currently missing links to making AI truly useful in finance, law, and government.
Future Outlook
The next frontier isn't just "more tokens"; it's multi-modal grounded reasoning. Until agents can "see" a chart and "understand" that a 1945 table has been superseded by a 1948 correction, they will remain "intern-level" assistants rather than "analyst-level" experts.