DOTA: Deformable Optimized Transformer Architecture for End-to-End Text Recognition with Retrieval-Augmented Generation

2025-05-07

AI Summary

Scene text recognition remains a fundamental challenge in computer vision and multimodal understanding. This paper proposes an end-to-end OCR framework that enhances geometric robustness by embedding deformable convolutions into the third and fourth stages of a ResNet-ViT hybrid backbone. To jointly model structural constraints and semantic context, it introduces, for the first time, a synergistic integration of retrieval-augmented generation (RAG) and conditional random fields (CRFs). Additionally, an adaptive dropout mechanism is designed to improve generalization. Evaluated on six standard benchmarks—including IC13 and IC15—the framework achieves a mean accuracy of 77.77%, with 97.32% on IC13—setting new state-of-the-art results across multiple datasets. The proposed method significantly advances recognition accuracy and robustness in complex, real-world scenes.

Technology Category

• Computer Vision: Scene Analysis & Understanding
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Text recognition in natural images remains a challenging yet essential task, with broad applications spanning computer vision and natural language processing. This paper introduces a novel end-to-end framework that combines ResNet and Vision Transformer backbones with advanced methodologies, including Deformable Convolutions, Retrieval-Augmented Generation, and Conditional Random Fields (CRF). These innovations collectively enhance feature representation and improve Optical Character Recognition (OCR) performance. Specifically, the framework substitutes standard convolution layers in the third and fourth blocks with Deformable Convolutions, leverages adaptive dropout for regularization, and incorporates CRF for more refined sequence modeling. Extensive experiments conducted on six benchmark datasets IC13, IC15, SVT, IIIT5K, SVTP, and CUTE80 validate the proposed method's efficacy, achieving notable accuracies: 97.32% on IC13, 58.26% on IC15, 88.10% on SVT, 74.13% on IIIT5K, 82.17% on SVTP, and 66.67% on CUTE80, resulting in an average accuracy of 77.77%. These results establish a new state-of-the-art for text recognition, demonstrating the robustness of the approach across diverse and challenging datasets.

Problem

• Enhancing text recognition in natural images using deformable convolutions and transformers
• Improving OCR performance with retrieval-augmented generation and CRF modeling
• Achieving state-of-the-art accuracy across diverse benchmark datasets

Innovation

• Deformable Convolutions enhance feature representation
• Retrieval-Augmented Generation improves OCR performance
• Conditional Random Fields refine sequence modeling