Visual Text Processing: A Comprehensive Review and Unified Evaluation

2025-04-30

AI Summary

This paper addresses the insufficient modeling and fusion of textual characteristics in vision-text processing. To tackle this, we propose a systematic solution: (1) We introduce VTPBench, the first end-to-end benchmark covering detection, recognition, reconstruction, and editing tasks; (2) We design VTPScore, an MLLM-based, semantics-aware automatic evaluation metric enabling fair, cross-task quantitative assessment; and (3) Through empirical analysis of over 20 state-of-the-art models, we identify pervasive deficiencies in semantic consistency and geometric fidelity. Our contributions are threefold: (1) The first unified, full-spectrum evaluation benchmark for vision-text processing tasks; (2) The first MLLM-driven evaluation framework explicitly supporting semantic understanding; and (3) Open-sourced, reproducible diagnostic tools and resources that provide both theoretical foundations and practical paradigms for text-specific modeling.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visual text is a crucial component in both document and scene images, conveying rich semantic information and attracting significant attention in the computer vision community. Beyond traditional tasks such as text detection and recognition, visual text processing has witnessed rapid advancements driven by the emergence of foundation models, including text image reconstruction and text image manipulation. Despite significant progress, challenges remain due to the unique properties that differentiate text from general objects. Effectively capturing and leveraging these distinct textual characteristics is essential for developing robust visual text processing models. In this survey, we present a comprehensive, multi-perspective analysis of recent advancements in visual text processing, focusing on two key questions: (1) What textual features are most suitable for different visual text processing tasks? (2) How can these distinctive text features be effectively incorporated into processing frameworks? Furthermore, we introduce VTPBench, a new benchmark that encompasses a broad range of visual text processing datasets. Leveraging the advanced visual quality assessment capabilities of multimodal large language models (MLLMs), we propose VTPScore, a novel evaluation metric designed to ensure fair and reliable evaluation. Our empirical study with more than 20 specific models reveals substantial room for improvement in the current techniques. Our aim is to establish this work as a fundamental resource that fosters future exploration and innovation in the dynamic field of visual text processing. The relevant repository is available at https://github.com/shuyansy/Visual-Text-Processing-survey.

Problem

• Identifying optimal textual features for diverse visual text tasks
• Integrating distinctive text features into processing frameworks effectively
• Developing fair evaluation metrics for visual text processing models

Innovation

• Leveraging foundation models for text processing
• Introducing VTPBench benchmark for datasets
• Proposing VTPScore for fair evaluation