Why parsing benchmarks can mislead you about real-world performance
Most public document parsing benchmarks report a single OCR or markdown-level similarity score, often on clean academic layouts or synthetic prose. That kind of metric correlates poorly with what downstream agents actually need: the correct value for a specific field on a messy real-world page. RealDocBench [3] shows that single-number benchmarks hide a wide performance spread across systems, and that a persistently hard medical sub-domain exists—something a single score would mask.
The implication is that a high benchmark score doesn't guarantee your system will extract the right field value from a blurry photo of a claim form. Instead, you need to evaluate on your actual document types and measure field-level accuracy, not just a similarity score.
Latency and cost are the metrics that make or break a production system
In real deployments, speed and cost per page are often more important than a marginal gain in parsing accuracy. HunyuanOCR-1.5 [1] demonstrates that a lightweight OCR VLM can achieve a 6.37x Transformer inference speedup and a 2.14x speedup under vLLM, making it the fastest among lightweight OCR VLMs. That speedup directly translates to lower latency and lower serving cost, which are critical for high-volume applications.
Similarly, a hybrid system using a compact VLM (Qwen 2.5-VL-7B) achieved an average processing latency of under 2 seconds per document while processing tens of thousands of claims weekly [2]. That's a 300x improvement over manual processing, which takes about 10 minutes per claim. These numbers show that deployment metrics like latency and throughput are what determine whether a system can handle real-world scale.
Field-level accuracy is the metric that matters for real documents
For regulated documents like mortgages, financial reports, and clinical records, the goal is not to reproduce the page but to extract the correct value for a specific field. RealDocBench [3] introduces a QA track with 1,356 field-level questions over 581 documents, scored on per-field and strict per-question accuracy. This approach reveals a wide performance spread that single-number benchmarks hide, and it shows that some domains (like medical) are persistently harder.
In a deployed claims-processing system, field-level extraction accuracy reached approximately 87% with a document-type classification accuracy of over 95% [2]. That level of accuracy is what makes automation viable in production. So, when comparing compact VLMs for document parsing, prioritize field-level accuracy on your own document types over a generic parsing benchmark score.
About These Sources
This answer is built on 3 studies (2 peer-reviewed, 1 preprint) — published in 2026, 3 from 2024 or later — selected as the most relevant from 3 studies that passed quality screening, drawn from 50 papers retrieved from a database of over 500 million.
Sources used in this answer
HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better
HunyuanOCR-1.5, a lightweight OCR VLM, achieves a 6.37x Transformer inference speedup and a 2.14x speedup under vLLM, making it the fastest among lightweight OCR VLMs, while maintaining top-tier performance on OmniDocBench.
A Hybrid Architecture for Multi-Stage Claim Document Understanding: Combining Vision-Language Models and Machine Learning for Real-Time Processing
A hybrid pipeline using a compact VLM (Qwen 2.5-VL-7B) achieved over 95% document-type classification accuracy, ~87% field-level extraction accuracy, and under 2 seconds average latency per document, processing tens of thousands of claims weekly in production.
RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents
RealDocBench, a benchmark with 1,356 field-level QA questions over 581 real regulated documents, shows that single-number benchmarks hide a wide performance spread and that field-level accuracy and cost/latency trade-offs are critical for real-world deployment.