Unlimited OCR: Simulating Human Working Memory for One-shot Long-horizon Parsing

TL;DR

Baidu’s Unlimited OCR breaks the "for-loop" bottleneck of document parsing. By introducing Reference Sliding Window Attention (R-SWA), the model can transcribe dozens of pages in a single forward pass. It eliminates the linear memory growth of traditional Transformers, achieving SOTA performance (93.92% on OmniDocBench) and a 35% speed increase over baseline models for long-form outputs.

Problem & Motivation: The Memory Trap of LLM-based OCR

The AI community has largely agreed that using Large Language Models (LLMs) as OCR decoders is the way forward—it allows the model to use linguistic priors to correct visual ambiguities. However, this comes with a "tax": the KV cache.

As the model generates text, the history it must remember grows linearly. For a document with 20 pages, the output might exceed 100k tokens, leading to:

1. Memory Exhaustion: The GPU runs out of VRAM to store the KV cache.
2. Latency Spikes: Each new token takes longer to generate than the last.

The authors observe that humans don't work this way. When you copy a book, you don't keep the first 10 pages in your active working memory as you write page 11; you only look at the source text (the reference) and the immediate previous context.

Methodology: Reference Sliding Window Attention (R-SWA)

The core innovation is the R-SWA mechanism. Unlike standard Multi-Head Attention (MHA) where every token looks at every previous token, R-SWA splits the attention into two zones:

1. Global Reference Zone ($m$): All visual tokens from the encoder and the prompt remain permanently accessible.
2. Causal Sliding Window ($n$): The decoder only "remembers" the last $n$ (default 128) generated tokens.

This architecture ensures that the KV cache is upper-bounded by $L_m+n$. No matter if you are transcribing the 100th or the 10,000th token, the memory footprint and the computational cost per token remain identical.

The "Free Lunch" Effect

Interestingly, the authors found that R-SWA isn't just a compression trick—it actually improves accuracy. Standard full attention can lead to "divergence" or confusion in dense OCR tasks as the sequence grows. By forcing the model to focus only on the current local context and the visual reference, the "inductive bias" of the model aligns better with the task of parsing.

Experiments & Results: Performance at Scale

Unlimited OCR was tested against a range of heavyweights including GPT-4o, Qwen2.5-VL, and the original DeepSeek OCR.

1. SOTA Benchmarking

On OmniDocBench v1.5, Unlimited OCR delivered a massive jump:

• Overall Score: 93.23 (+6.22 over DeepSeek OCR baseline).
• Table Extraction (TEDS): Improved by nearly 6%, showing that local causal context is sufficient for structural understanding.

2. Efficiency Gains

In terms of Throughput (TPS), the advantage of Unlimited OCR scales with the length of the document.

As shown above, the latency for DeepSeek OCR (MHA) spikes and grows, while Unlimited OCR remains flat. At 6,144 tokens, Unlimited OCR is 35% faster than the baseline.

3. Long-Horizon Stress Test

The model successfully parsed documents exceeding 40 pages in one shot. Even at the 40+ page mark, the Edit Distance remained as low as 0.1069, proving that the "soft forgetting" mechanism didn't lose the "reading track."

Critical Analysis & Conclusion

Takeaway: Unlimited OCR proves that we don't need infinite memory to handle infinite tasks. By mimicking the human ability to prioritize reference material over historical output, Baidu has created a model that is both faster and more accurate for document intelligence.

Limitations: The model is still limited by the prefill length of the initial image tokens. While the output memory is constant, the input memory (visual tokens) still grows with the number of pages. The authors suggest that future work will involve "automatic page flipping" (dynamic prefill fetching) to overcome the 32k/128k context limits.

Future Outlook: R-SWA is a "General-purpose Parsing Attention." Expect to see this logic ported to ASR (Speech) and Translation, where the "source" is long but the "working memory" requirements are local.