SwinTextSpotter v2: Towards Better Synergy for Scene Text Spotting

2024-01-15

arXiv.org

AI Summary

Existing end-to-end scene text recognition methods share only the backbone network, lacking deep feature interaction between detection and recognition tasks, thereby limiting recognition accuracy. To address this, we propose the first bi-directional explicit detection–recognition collaboration framework: Recognition Conversion enables recognition loss to guide localization in reverse, while Recognition Alignment dynamically aligns detection predictions with recognition features for cross-task mutual enhancement. We further introduce a Box Selection Schedule strategy to significantly reduce detector parameter count. Built upon the Swin Transformer backbone, our method requires neither character-level annotations nor explicit rectification modules. It supports arbitrary-shape and multilingual (e.g., English, Chinese, Vietnamese) text recognition. On mainstream curved-text benchmarks, it achieves state-of-the-art performance with a lighter model and weaker supervision.

Technology Category

• Computer Vision: Object Detection & Categorization
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

End-to-end scene text spotting, which aims to read the text in natural images, has garnered significant attention in recent years. However, recent state-of-the-art methods usually incorporate detection and recognition simply by sharing the backbone, which does not directly take advantage of the feature interaction between the two tasks. In this paper, we propose a new end-to-end scene text spotting framework termed SwinTextSpotter v2, which seeks to find a better synergy between text detection and recognition. Specifically, we enhance the relationship between two tasks using novel Recognition Conversion and Recognition Alignment modules. Recognition Conversion explicitly guides text localization through recognition loss, while Recognition Alignment dynamically extracts text features for recognition through the detection predictions. This simple yet effective design results in a concise framework that requires neither an additional rectification module nor character-level annotations for the arbitrarily-shaped text. Furthermore, the parameters of the detector are greatly reduced without performance degradation by introducing a Box Selection Schedule. Qualitative and quantitative experiments demonstrate that SwinTextSpotter v2 achieved state-of-the-art performance on various multilingual (English, Chinese, and Vietnamese) benchmarks. The code will be available at href{https://github.com/mxin262/SwinTextSpotterv2}{SwinTextSpotter v2}.

Problem

• Improves synergy between text detection and recognition
• Eliminates need for rectification modules or character-level annotations
• Reduces detector parameters without performance loss

Innovation

• Recognition Conversion guides text localization
• Recognition Alignment extracts text features dynamically
• Box Selection Schedule reduces detector parameters