Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition

2026-08-10

AI Summary

This work addresses the significant performance drop of Optical Chemical Structure Recognition (OCSR) when transitioning from synthetic images to real-world documents—a manifestation of the synthetic-to-real domain gap. The study systematically investigates the impact of fine-tuning vision-language models with hybrid datasets combining synthetic and real annotated data. It provides the first quantitative evidence that real data plays a decisive role in bridging this domain gap and reveals that the efficacy of visual adapter strategies such as LoRA is highly dependent on the choice of base model. Experiments across Qwen2.5-VL, InternVL3-8B, and GLM-4.1V-9B demonstrate that optimal configurations achieve exact match scores of 0.49, 0.65, 0.84, and 0.76 on the ACS, CLEF-IP, UOB, and USPTO real-world benchmarks, respectively, substantially narrowing the performance disparity with the synthetic domain.

Abstract

Millions of chemical structures appear in patents and papers only as drawings, and using that information at scale requires reading the drawings. OCSR appears nearly solved on synthetic images yet remains difficult on real documents: the starting recognizer, Qwen2.5-VL-7B, exceeds 91% accuracy on synthetic renders but falls below 16% on three real-world benchmarks (ACS, CLEF-IP, USPTO). To identify the main source of improvement, 21 recognizers were fine-tuned on mixtures of synthetically rendered structures and labeled real depictions from patents, journal figures, and hand-drawn collections, varying the vision language model (VLM) base, the fraction of real training data, and the vision-tower adaptation strategy. Labeled real training images make the largest difference. For Qwen2.5-VL, ACS exact match rises from 0.15 with no real data to 0.37 at 9.5% and 0.46 at 50.2%; a controlled experiment across three base models reproduces the trend. A vision-tower LoRA, in contrast, does nothing for Qwen (+0.00, paired p=1.00), substantially helps InternVL3-8B (+22.8 to +34.6 pt), and modestly helps GLM-4.1V-9B (+1.0 to +9.6 pt), so its value depends on the base model. The best configuration reaches 0.96 exact match on clean renders and 0.49, 0.65, 0.84, and 0.76 on ACS, CLEF-IP, UOB, and USPTO, respectively. Gaps between base models are largest without real data (0.21), shrink to 0.06 at 70% real data, and reorder the ranking; base model and real-data mixture must therefore be selected together. Small-scale experiments on handwritten image-to-LaTeX recognition and chart-to-table conversion show that base-model rankings also vary beyond chemistry. More generally, model and adaptation choices for visual structure recognition should be evaluated on the target task.

Problem

• Optical Chemical Structure Recognition
• synthetic-to-real gap
• real-world document understanding
• vision-language models
• chemical structure extraction

Innovation

• optical chemical structure recognition
• synthetic-to-real gap
• real-world data fine-tuning
• vision language models
• LoRA adaptation