[CVPR 2025] FireRed-OCR: Taming Structural Hallucinations in VLMs with Format-Constrained GRPO

TL;DR

FireRed-OCR, developed by Xiaohongshu's Super Intelligence Team, is a systematic framework designed to bridge the gap between "seeing text" and "understanding structure." By evolving a 2B-parameter Qwen3-VL model through a specialized three-stage pipeline—culminating in Format-Constrained Group Relative Policy Optimization (GRPO)—it has set a new SOTA on the OmniDocBench v1.5 with a score of 92.94%. It proves that targeted RL can make a small model outperform giants like GPT-4o and Gemini 1.5 Pro in complex document parsing.

Background: The Crisis of Structural Hallucination

While modern Vision-Language Models (VLMs) are exceptional at describing scenes, they are notoriously "sloppy" engineers. In industrial OCR tasks, a single missing pipe character | in a Markdown table or a mismatched brace { in LaTeX makes the output unusable for downstream automation. The authors term this Structural Hallucination: the model gets the words right but the "scaffolding" wrong.

The fundamental challenge isn't just recognition; it's behavioral alignment. General VLMs aren't incentivized to care about syntax closure or row-column counts. FireRed-OCR addresses this by treating document parsing as a constrained optimization problem.

Methodology: The Three-Stage Evolution

The authors propose a "coarse-to-fine" curriculum to transform a generalist into a specialist.

1. The "Geometry + Semantics" Data Factory

Before training, the authors solved the "Garbage In, Garbage Out" problem. Instead of random sampling, they used Geometric Clustering to identify and upsample "long-tail" layouts (e.g., nested tables, multi-column scientific papers). They also employed Expert-Level Refinement, using frontier models (like Gemini Pro) to distill high-quality logic into their training set.

2. The Training Pipeline

The model follows a three-stage progressive training strategy:

• Stage 1: Multi-task Pre-alignment: Grounding the model by forcing it to predict bounding boxes alongside text.
• Stage 2: Specialized SFT: Refining the model on 400k high-quality pairs to standardize Markdown output.
• Stage 3: Format-Constrained GRPO: The "secret sauce." Using Reinforcement Learning to reward specific structural traits.

Figure 1: The Three-Stage Training Pipeline, highlighting the transition from spatial grounding to reinforcement learning.

Deep Insight: Why GRPO for OCR?

The most innovative aspect is the use of GRPO. Unlike PPO, which requires a heavy Value model, GRPO estimates the baseline from a group average of outputs.

The authors designed four specific rewards:

1. Formula Syntax: Does the LaTeX compile?
2. Hierarchical Closure: Are all Markdown tags closed?
3. Table Integrity: Does every row have the same number of columns?
4. Textual Accuracy: Is the content faithful to the original image?

This allows the model to "self-correct" during the RL phase, learning that a syntactically perfect table is more valuable than a semantically similar but broken one.

Experiments and Results

FireRed-OCR-2B was tested against the industry's best. On OmniDocBench v1.5, it achieved a dominant 92.94%, beating DeepSeek-OCR 2 (91.09%) and even the gargantuan Qwen3-VL-235B (89.15%).

Quantitative SOTA

Figure 2: Benchmark results on OmniDocBench v1.5. FireRed-OCR leads across all sub-metrics.

Qualitative Excellence

The model's ability to handle complex 2D structures is particularly evident in mathematical parsing and financial tables, where it successfully avoids the "row-shift" errors common in other E2E models.

Figure 3: Complex layout analysis. The model correctly parses hierarchical headers and spanning cells in financial reports.

Critical Analysis & Conclusion

The success of FireRed-OCR sends a clear signal: Efficiency > Scale. By focusing on the distribution of data and the constraints of the output, a 2B model can effectively replace a 200B+ model for specialized tasks.

Limitations: While the model is a structural expert, the authors noted some "modality interference" when trying to optimize for tables and text simultaneously. Their solution—a Balanced Mixture Strategy—was necessary to prevent one domain from degrading the other.

Future Outlook: The "General VLM to Structural Expert" paradigm via GRPO is likely to be adopted for other rigorous tasks like code generation and medical report parsing.

Takeaway

FireRed-OCR isn't just another OCR model; it’s a blueprint for taming the "creativity" of LLMs into the "precision" required for industrial applications.

For more details, check out the GitHub Repository.