SAViL-Det: Semantic-Aware Vision-Language Model for Multi-Script Text Detection

2025-07-27

AI Summary

Natural scene text detection faces challenges including difficulty in recognizing multilingual scripts and arbitrarily shaped (e.g., curved) text, as well as insufficient semantic richness in visual cues. To address these, we propose a semantic-aware vision-language detection framework: it leverages the CLIP pre-trained model and an Asymptotic Feature Pyramid Network (AFPN) to construct multi-scale visual representations; introduces a text-to-pixel contrastive learning mechanism and a language-vision decoder that employs cross-attention for fine-grained cross-modal semantic alignment. This work is the first to incorporate strong linguistic semantics into end-to-end text detection, significantly enhancing robustness for complex scripts and curved text. Our method achieves state-of-the-art F-scores of 84.8% on MLT-2019 and 90.2% on CTW1500, surpassing prior approaches.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Detecting text in natural scenes remains challenging, particularly for diverse scripts and arbitrarily shaped instances where visual cues alone are often insufficient. Existing methods do not fully leverage semantic context. This paper introduces SAViL-Det, a novel semantic-aware vision-language model that enhances multi-script text detection by effectively integrating textual prompts with visual features. SAViL-Det utilizes a pre-trained CLIP model combined with an Asymptotic Feature Pyramid Network (AFPN) for multi-scale visual feature fusion. The core of the proposed framework is a novel language-vision decoder that adaptively propagates fine-grained semantic information from text prompts to visual features via cross-modal attention. Furthermore, a text-to-pixel contrastive learning mechanism explicitly aligns textual and corresponding visual pixel features. Extensive experiments on challenging benchmarks demonstrate the effectiveness of the proposed approach, achieving state-of-the-art performance with F-scores of 84.8% on the benchmark multi-lingual MLT-2019 dataset and 90.2% on the curved-text CTW1500 dataset.

Problem

• Detecting multi-script text in natural scenes
• Integrating textual prompts with visual features
• Improving text detection for diverse and curved scripts

Innovation

• Integrates textual prompts with visual features
• Uses CLIP and AFPN for feature fusion
• Employs text-to-pixel contrastive learning