[arXiv 2026] Beyond Strings and Grids: LLM-based Semantic Evaluation Redefines PDF Table Extraction Benchmarking

TL;DR

Table extraction remains one of the "boss levels" of document AI. This paper argues that our current ways of measuring success—specifically metrics like TEDS and GriTS—are fundamentally broken because they care too much about HTML tags and not enough about data integrity. By introducing an LLM-as-a-judge framework and a synthetic benchmark of 451 tables, the authors prove that LLMs can evaluate table quality with 93% human correlation, revealing that even "SOTA" parsers often hallucinate critical numerical data.

The "Representational Gap" Problem

Why is a table with a missing decimal point (112 instead of 1.12) often scored higher by traditional metrics than a table that simply uses Markdown instead of HTML?

Current metrics are syntactic, not semantic. They treat a table as a tree or a grid of strings. If a parser flattens a header for compatibility reasons, it gets penalized heavily, even if the information is perfectly preserved. Conversely, a parser that maintains a perfect grid but flips a "+" to a "-" receives a near-perfect score. This lack of semantic awareness makes existing leaderboards unreliable for high-stakes fields like finance or medicine.

Methodology: Synthetic Ground Truth & LLM Judges

The authors solved the evaluation crisis with a two-pronged approach:

1. Synthetic PDF Generation: Instead of manually labeling PDFs (which is error-prone), they took raw LaTeX code from arXiv, cleaned it, and compiled it into fresh PDFs. This provides a "pixel-perfect" ground truth in LaTeX format.
2. The LLM-as-a-Judge Pipeline: They replaced rigid string-matching algorithms with a multimodal LLM. The judge is asked: "Considering headers and cell values, is the information in the parsed table identical to the ground truth?"

Insights from the Leaderboard (21 Parsers)

The study benchmarked 21 tools, including specialized OCR models (Mathpix, MinerU), general VLMs (Gemini, Claude), and rule-based tools (PyMuPDF).

Key Findings:

• The SOTA Leader: Gemini 3 Pro and Flash dominated the field, proving that general-purpose multimodal reasoning is currently superior to specialized table-extraction models.
• The Underdog: LightOnOCR-2-1B (a 1-billion parameter model) punched way above its weight class, outperforming much larger models like Qwen3-VL and GPT-5 mini.
• Rule-based Failure: Tools like GROBID and PyMuPDF4LLM are essentially obsolete for complex scientific tables, scoring as low as 2/10.

The scatter plots above clearly show that LLM judges (right) cluster much more tightly with human scores than rule-based metrics (left).

Performance Distribution: The "Bimodal" Trap

One of the paper’s most interesting insights is the parser failure pattern. Some models, like Claude Sonnet 4.6, exhibit a bimodal distribution: they either extract the table perfectly (Score 10) or fail to see it at all (Score 0). Other models provide a "mediocre middle," where the table is always detected but frequently contains subtle, dangerous errors in the data.

Critical Analysis & Future Outlook

While the use of LLMs as judges solves the semantic correlation problem, it introduces proprietary dependency. Relying on Gemini or Claude to evaluate other models is expensive and creates a "black box" evaluation layer.

However, the takeaway is clear: If you are building a RAG pipeline or a knowledge graph, stop using TEDS to pick your parser. This paper provides a reproducible framework to choose a parser based on what actually matters—the accuracy of the underlying data.

Future Work

The authors suggest that the next frontier is integrated parsing—evaluating how well models handle the intersection of tables, surrounding text, and inline formulas simultaneously, rather than treating tables as isolated objects.