Can lightweight open-source vision-language models extract structured JSON from heritage documents?

The task is OCR plus schema-bound JSON

The paper studies OCR-to-structure: given an image of an archival document, a model must read the text and return it as a JSON object that follows the collection's own field schema. That second half matters. A model can transcribe characters well and still fail if it puts a value under the wrong key, merges fields, drops a field or invents an extra one.

The authors make this distinction concrete with three metrics. Character Error Rate (CER) measures how far the extracted text is from the reference text, with lower being better. Approximate Normalized Levenshtein Similarity (ANLS*) scores string similarity, with higher being better. Mean Average Precision F1 (mAP-F1) scores whether values land in the correct fields of the structured output. The paper notes that a model can show low CER and high ANLS* yet still get a low mAP-F1 when field assignment goes wrong, so the three metrics answer different questions and should not be read as substitutes.

Eight models, each tested through its own interface

The comparison covers eight open-source vision-language models ranging from 200M to 7B parameters, spanning encoder-decoder, unified vision-language, OCR-specialized and instruction-tuned architectures. The data are three university heritage collections: Erlangen-Prehistoric (308 documents, handwritten, 8 fields), Pforzheim Jewelry (590 documents, mostly printed, 17 fields) and Regensburg Scalalogy (164 documents, mixed text, nested schema with booleans). All are split 70% training, 15% validation and 15% test, for 1,062 documents in total.

A central methodological choice is the constraint-aware protocol. Donut-Base, Donut-CORD, Florence-2-Large and PaddleOCR-VL accept only fixed OCR-style prompts or task-specific tokens, so they cannot take natural-language instructions or few-shot demonstrations. Nanonets-OCR-s, Gemma-3-4B, Phi-3.5-Vision and Qwen2.5-VL-7B accept custom prompts, which allows schema-guided extraction. Each model is therefore evaluated through the interface it actually supports, with minimal post-processing of raw generations before JSON parsing. This design keeps the comparison fair to each model's real deployment options, but it also means the fixed-prompt models are not being asked for structured output in the same way as the prompt-flexible ones.

Zero-shot output separates text readers from structure writers

Without any examples, the fixed-prompt models largely failed at the structured task. Donut-Base and Donut-CORD returned 0.0 ANLS* and 0.0 mAP-F1 on all three datasets, and Florence-2-Large and PaddleOCR-VL also scored 0.0 mAP-F1 throughout. Their CER values were not uniformly terrible, which illustrates the metric split: readable text did not translate into parseable, schema-conformant JSON.

Among prompt-flexible models, Qwen2.5-VL-7B reached the best zero-shot CER on Erlangen-Prehistoric (21.4%) and the highest mAP-F1 there (56.2%), while on Pforzheim Jewelry Gemma-3-4B and Qwen2.5-VL-7B both produced high ANLS* (92.1% and 96.0%) and mAP-F1 above 87%. Regensburg Scalalogy, the nested and structurally complex collection, was harder for everyone: the best zero-shot mAP-F1 was 33.3% from Qwen2.5-VL-7B. The authors read this pattern as evidence that reliable extraction depends not only on recognizing text but also on following instructions, binding values to fields and decoding a stable schema.

Fine-tuning is the most reliable single improvement

Fine-tuning changed the picture substantially. Qwen2.5-VL-7B reached 5.8% CER, 91.0% ANLS* and 90.0% mAP-F1 on Erlangen-Prehistoric, 0.5% CER and 99.4% mAP-F1 on Pforzheim Jewelry, and 7.6% CER with 62.8% mAP-F1 on Regensburg Scalalogy. Phi-3.5-Vision reached 99.5% mAP-F1 on Pforzheim Jewelry and 72.0% on Regensburg Scalalogy, and Gemma-3-4B reached 99.4% and 46.9% on those two collections. The authors select these three as the top models because they combine low CER, high ANLS* and high mAP-F1 with near-saturated JSON validity of roughly 95% to 100% across settings.

The fine-tuning setup is part of the practical claim. Models below 1B parameters were fully fine-tuned, while larger models used QLoRA with 4-bit quantization and LoRA adapters on attention and MLP layers, keeping base weights frozen on a single NVIDIA V100 with 32GB of VRAM. The best checkpoint was chosen by lowest validation CER. That matters for institutions: the paper's case for local deployment rests on adaptation that fits one consumer-grade GPU, not on full retraining of large models.

Fine-tuning did not rescue every architecture. Donut models could reach lower CER and higher ANLS* after fine-tuning yet still scored 0.0 mAP-F1 because their outputs often failed strict JSON dictionary parsing. Text accuracy and schema compliance remained separate outcomes.

Few-shot examples help printed records more than handwriting

Few-shot prompting was tested only on the four prompt-flexible models, using one demonstration for Pforzheim Jewelry and two for the other two collections. The results were mixed. On Pforzheim Jewelry, Qwen2.5-VL-7B reached 0.9% CER, 98.9% ANLS* and 99.4% mAP-F1, and Gemma-3-4B reached 2.0% CER and 92.1% mAP-F1, so examples worked well on cleaner printed records.

On the handwritten and structurally complex collections the picture reversed or blurred. Nanonets-OCR-s improved its Erlangen-Prehistoric mAP-F1 from 28.5% zero-shot to 44.7% few-shot, but Phi-3.5-Vision's CER on Regensburg Scalalogy rose to 52.7% and its mAP-F1 fell to 23.1%. The authors attribute the inconsistency to longer prompts increasing instruction complexity on harder documents. Few-shot prompting is therefore not a dependable substitute for fine-tuning in this benchmark.

Preprocessing and multi-stage training are not safe defaults

The paper isolates three additional strategies on the top three models. Hyperparameter optimization, run with Optuna over 20 trials, gave the most consistent modest gains, with the largest improvements on Regensburg Scalalogy: Gemma-3-4B's mAP-F1 rose from 46.9% to 77.9% and its CER fell from 14.6% to 3.4%, while Qwen2.5-VL-7B's CER fell from 7.6% to 3.1%. Even here the effect was uneven; on Erlangen-Prehistoric, Gemma-3-4B's mAP-F1 dropped from 83.1% to 65.4% despite other metrics improving.

Classical image preprocessing, combining illumination flattening, denoising, CLAHE and letterbox resizing, was mixed and often harmful. It lowered Phi-3.5-Vision's Regensburg Scalalogy CER from 6.3% to 5.5% but cut its mAP-F1 there from 72.0% to 61.1%; it also raised Qwen2.5-VL-7B's Regensburg CER from 7.6% to 12.0%. Multi-stage curriculum training consistently helped Gemma-3-4B across all three datasets, raising Regensburg mAP-F1 from 46.9% to 63.8%, but degraded Qwen2.5-VL-7B on Pforzheim Jewelry and Regensburg Scalalogy. The authors treat both as empirical enhancements to be tested per model and dataset rather than assumed wins.

One shared checkpoint helps Gemma but hurts Qwen

The deployment question the paper closes on is whether an institution should keep one checkpoint per collection or train a single checkpoint on the union of all training data. The answer depends on the model. Gemma-3-4B improved under joint training on all three collections, with Regensburg Scalalogy mAP-F1 rising from 46.9% to 68.0% and CER falling from 14.6% to 9.1%, which the authors read as better cross-schema robustness.

Qwen2.5-VL-7B moved the other way. Its Regensburg Scalalogy CER rose from 7.6% to 24.9% and ANLS* fell from 87.0% to 65.2%, with mAP-F1 dropping from 62.8% to 41.4%; Pforzheim Jewelry also worsened slightly. Phi-3.5-Vision degraded on all three datasets, though less severely, with Erlangen-Prehistoric mAP-F1 falling from 82.2% to 45.4%. The paper therefore finds no general rule for multi-dataset training: it can improve cross-collection robustness or introduce negative transfer depending on the model.

The practical priority that follows is model-specific validation before consolidation. An institution weighing one shared checkpoint against separate per-collection checkpoints should test both on its own schemas, because the same joint-training recipe that helped Gemma-3-4B hurt Qwen2.5-VL-7B most on the nested Regensburg collection. The authors also note two limits on scope: the three improvement strategies were evaluated independently rather than in combination, and the lightweight VLM landscape changes quickly, so the findings are guidance for the evaluated model families rather than a statement about all emerging document models.

Sources used in this answer