🤖 AI Summary
This work proposes a billion-parameter, end-to-end multilingual vision-language model that directly converts document images into well-structured, naturally ordered text while enabling precise localization of embedded images. Addressing the error-proneness and inefficiency of traditional multi-stage OCR pipelines in handling multilingual documents, the model introduces a novel curriculum-based bounding box localization strategy during pretraining. It further incorporates reinforcement learning with an IoU-based reward mechanism (RLVR) and enhances robustness through checkpoint averaging and task arithmetic fusion. Evaluated on OlmOCR-Bench, the model achieves state-of-the-art performance while being nine times smaller than the previous best model and offering significantly faster inference. The authors publicly release the model, training data, and a new evaluation benchmark, LightOnOCR-bbox-bench.
📝 Abstract
We present \textbf{LightOnOCR-2-1B}, a 1B-parameter end-to-end multilingual vision--language model that converts document images (e.g., PDFs) into clean, naturally ordered text without brittle OCR pipelines. Trained on a large-scale, high-quality distillation mix with strong coverage of scans, French documents, and scientific PDFs, LightOnOCR-2 achieves state-of-the-art results on OlmOCR-Bench while being 9$\times$ smaller and substantially faster than prior best-performing models. We further extend the output format to predict normalized bounding boxes for embedded images, introducing localization during pretraining via a resume strategy and refining it with RLVR using IoU-based rewards. Finally, we improve robustness with checkpoint averaging and task-arithmetic merging. We release model checkpoints under Apache 2.0, and publicly release the dataset and \textbf{LightOnOCR-bbox-bench} evaluation under their respective licenses.