An Effective Data Augmentation Method by Asking Questions about Scene Text Images

2026-03-03

AI Summary

This work proposes a visual question answering (VQA)-based data augmentation framework for optical character recognition (OCR), addressing the limitation of conventional OCR models that directly predict transcriptions without fine-grained reasoning about text structure. For the first time, structured VQA tasks are introduced into both scene and handwritten text recognition by automatically generating natural language questions concerning character existence, position, and frequency, with answers derived from ground-truth transcriptions. This approach encourages the model to jointly reason over visual and semantic information through multi-task learning, thereby enhancing character-level understanding and visual-text alignment. Experimental results demonstrate significant reductions in character error rate (CER) and word error rate (WER) on the WordArt and Esposalles datasets, outperforming existing baselines.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Question Answering
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Scene text recognition (STR) and handwritten text recognition (HTR) face significant challenges in accurately transcribing textual content from images into machine-readable formats. Conventional OCR models often predict transcriptions directly, which limits detailed reasoning about text structure. We propose a VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks. For each image-text pair, we generate natural-language questions probing character-level attributes such as presence, position, and frequency, with answers derived from ground-truth text. These auxiliary tasks encourage finer-grained reasoning, and the OCR model aligns visual features with textual queries to jointly reason over images and questions. Experiments on WordArt and Esposalles datasets show consistent improvements over baseline models, with significant reductions in both CER and WER. Our code is publicly available at https://github.com/xuyaooo/DataAugOCR.

Problem

• Scene Text Recognition
• Handwritten Text Recognition
• OCR
• Data Augmentation
• Text Transcription

Innovation

• data augmentation
• scene text recognition
• visual question answering
• handwritten text recognition
• structured reasoning