WeCromCL: Weakly Supervised Cross-Modality Contrastive Learning for Transcription-only Supervised Text Spotting

2024-07-28

European Conference on Computer Vision

AI Summary

This work addresses scene text detection and recognition under weak supervision—where only image–text pairs are provided, without any character- or word-level bounding box annotations. We propose a cross-modal contrastive learning framework that implicitly establishes fine-grained alignments between individual characters and their corresponding image regions. Crucially, our method introduces the first character-level atomic vision–language alignment mechanism: leveraging contrastive learning to localize character-level anchors without explicit spatial supervision, then generating high-quality pseudo-location labels to supervise an end-to-end differentiable text recognizer. Evaluated on four standard benchmarks—CTW1500, Total-Text, ICDAR2015, and SCUT-CTW1500—our approach significantly outperforms existing weakly supervised methods and even surpasses several fully supervised counterparts. To our knowledge, this is the first work to demonstrate the feasibility of achieving high-accuracy, character-level localization and recognition using purely transcription-level supervision.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Transcription-only Supervised Text Spotting aims to learn text spotters relying only on transcriptions but no text boundaries for supervision, thus eliminating expensive boundary annotation. The crux of this task lies in locating each transcription in scene text images without location annotations. In this work, we formulate this challenging problem as a Weakly Supervised Cross-modality Contrastive Learning problem, and design a simple yet effective model dubbed WeCromCL that is able to detect each transcription in a scene image in a weakly supervised manner. Unlike typical methods for cross-modality contrastive learning that focus on modeling the holistic semantic correlation between an entire image and a text description, our WeCromCL conducts atomistic contrastive learning to model the character-wise appearance consistency between a text transcription and its correlated region in a scene image to detect an anchor point for the transcription in a weakly supervised manner. The detected anchor points by WeCromCL are further used as pseudo location labels to guide the learning of text spotting. Extensive experiments on four challenging benchmarks demonstrate the superior performance of our model over other methods. Code will be released.

Problem

• Computer Vision
• Text Recognition
• Unsupervised Learning

Innovation

• WeCromCL
• unsupervised learning
• text localization