Don't sample randomly—sample the pages where OCR actually fails
If you evaluate OCR on a long document by picking a few random pages, you'll likely miss the real problems. The largest study here, Dr. DocBench, built a benchmark from a multilingual book corpus of documents averaging around 100 pages each, and deliberately selected the hardest pages by using parser-failure-based sampling—meaning they picked pages where multiple state-of-the-art OCR systems already struggled [1]. This approach revealed that strong performance on existing benchmarks does not transfer to expert-level content: systems that ace common documents fail on chemical formulas, music notation, complex tables, and cross-page layouts [1]. For your own evaluation, this means you should identify the most challenging sections of your ultra-long document—like dense tables, footnotes, or multi-column layouts—and focus your quality checks there, because that's where subjective quality issues will be most visible.
Use a human-annotated rubric to score quality, not just accuracy
Subjective quality isn't just about character accuracy—it's about whether the output is usable. The OCR-Quality dataset provides a practical model: it contains 1,000 PDF pages converted to images at 300 DPI, sampled from diverse real-world scenarios like academic papers, textbooks, and multilingual documents, and each page was manually annotated with a 4-level quality score: 1 (Excellent), 2 (Good), 3 (Fair), and 4 (Poor) [2]. This gives you a simple, repeatable scale to apply to your own long documents. The dataset also includes annotation guidelines and representative cases across difficulty levels, so you can calibrate your own evaluators or even train a model to predict quality [2]. The key takeaway: define what 'good' means for your use case (e.g., reading order preserved, tables intact, no garbled characters) and score pages against that rubric, rather than relying on a single accuracy number.
Evaluate at the page or block level, not just the whole document
Ultra-long documents often cover multiple topics, so a single overall quality score can hide serious problems in specific sections. The fine-grained distillation (FGD) framework for long-document retrieval addresses this by producing representations that are consistent across different granularities—from whole documents down to smaller segments—and then aligning them during training [3]. While FGD is about retrieval, the principle applies directly to OCR quality evaluation: you should assess quality at the page, block, or even table level, because a document can be 'good' overall but have a catastrophic error in one critical table or formula. Dr. DocBench supports this by providing block-level annotations for layout, reading order, and hierarchical relations, which lets you pinpoint exactly where quality breaks down [1]. In practice, this means you should generate quality scores for each page or section of your ultra-long document, then aggregate them to see which parts need attention.
About These Sources
This answer is built on 3 studies (all preprints) — published from 2022 to 2026, 2 from 2024 or later — selected as the most relevant from 3 studies that passed quality screening, drawn from 74 papers retrieved from a database of over 500 million.
Sources used in this answer
Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing
Dr. DocBench built a 4,514-page benchmark from long documents averaging ~100 pages, using parser-failure-based sampling to select hard pages, and found that state-of-the-art OCR systems fail on expert-level content like chemical formulas, music notation, and complex tables.
OCR-Quality: A Human-Annotated Dataset for OCR Quality Assessment
OCR-Quality provides 1,000 human-annotated PDF pages (converted to 300 DPI images) with a 4-level quality score (Excellent to Poor), including guidelines and representative cases, to support OCR quality assessment.
Fine-Grained Distillation for Long Document Retrieval
Fine-grained distillation (FGD) for long-document retrieval addresses the granular-mismatch issue by aligning representations across different granularities, achieving state-of-the-art performance on two long-document retrieval benchmarks.