[ICLR 2025] ExtractBench: Exposing the "Breadth Collapse" of Frontier LLMs in Structured Extraction

TL;DR

ExtractBench is a new stress-test for document AI, moving beyond simple entity extraction to enterprise-scale PDF-to-JSON parsing. The findings are a wake-up call for the industry: despite their hype, frontier models like GPT-5, Gemini 3, and Claude 4.5 fail completely (0% success) when asked to populate large schemas (369 fields), proving that schema breadth is the new frontier for LLM reliability.

Context: The Million-Dollar Extraction Problem

Enterprises are desperate to turn unstructured PDFs (contracts, financial filings, medical records) into clean, machine-readable JSON. While we’ve made progress in Long-Context windows, the industry's dirty secret is that we lack a principled way to evaluate if a model actually filled out a massive JSON schema correctly. Most benchmarks use tiny schemas (3-10 fields); ExtractBench introduces schemas with hundreds of fields across 12,867 evaluatable points.

Why Current Evaluations Fail

The authors identify two critical gaps:

1. Scale Gap: Real world schemas are "wide" and "deep." Most benchmarks use clean text prompts, but real extraction happens from messy PDFs.
2. Methodology Gap: How do you score a financial report? You can't use Exact Match for a long company name, but you must use it for a Tax ID. You need tolerance for numbers and semantic alignment for lists of citations.

The Core Innovation: Schema-Driven Evaluation

Instead of a single global F1 score, ExtractBench treats the JSON Schema as an executable specification. Inside the schema, developers define the evaluation_config for each leaf node:

• Identifiers: string_exact
• Financial Values: number_tolerance (e.g., 0.1% margin)
• Descriptions: string_semantic (LLM-based judging)
• Arrays: array_llm (Semantic alignment to handle reordering)

Figure 1: Complex financial schema structure showcasing nesting and multi-type leaf definitions.

Shocking Results: The Frontier Model Baseline

The researchers tested the "Big Three": OpenAI, Google, and Anthropic. The results were sobering.

1. The Breadth Cliff

On the SEC 10-K/Q domain (369 fields), the valid JSON rate was 0% across the board. Models simply could not maintain structural integrity or stay within token limits when the output required ~24k tokens of precisely formatted JSON.

2. Output Volume vs. Input Length

A fascinating insight: Credit Agreements are massive (~137 pages) but have simple schemas (13 keys). Models were surprisingly good here (>85% pass rate). Conversely, Research Papers are shorter but require enumerating hundreds of citations. Models failed here frequently. The conclusion: Output complexity, not input context, is the primary failure driver.

Figure 2: Credit Agreements represent "needle-in-a-haystack" (high compression), whereas Research Papers require massive output volume (low compression).

3. The "Structured Output" Paradox

Ironically, using "Structured Output" modes (which use constrained decoding/grammars) often decreased accuracy. For very complex schemas, the providers' internal grammar compilers often rejected the schema entirely or "distracted" the model from the content processing, leading to lower pass rates than raw prompting.

Critical Insight & Limitations

ExtractBench proves that nesting depth isn't the problem—schema breadth and array density are. If you are building a production RAG or extraction pipeline, these results suggest you cannot simply "throw a PDF and a huge schema" at an LLM and expect reliable JSON.

Future Outlook: The community needs to move toward "Decomposition Strategies"—breaking large schemas into smaller agentic tasks—rather than waiting for a single model to solve 400-field extractions in one shot.

Summary Table of Performance

Table 1: Per-domain results showing the total collapse of performance on the SEC 10-K/Q (369 keys) task.

ExtractBench is open-source at https://github.com/ContextualAI/extract-bench.