Visual Perception in Text Strings

2024-10-02

arXiv.org

AI Summary

This study investigates the capability of large language models (LLMs) and multimodal large language models (MLLMs) to perceive visual semantics in ASCII art—a novel modality-agnostic proxy task for visual understanding. We introduce the first dual-modality (text + image) classification benchmark specifically designed for ASCII art recognition, accompanied by a hierarchical evaluation framework and a supervised fine-tuning pipeline. Experimental results reveal a fundamental deficiency in cross-modal fusion: current MLLMs show no performance gain when jointly processing textual and ASCII-image inputs. While GPT-4o achieves 82.68% accuracy with image input—outperforming the strongest open-source MLLM by +21.95%—the best model under pure text input attains only ~30%, far below human performance (~100%). This work is the first systematic demonstration that ASCII art serves as an effective, lightweight, and interpretable probe for assessing visual-semantic understanding, establishing a new paradigm for efficient and transparent multimodal capability evaluation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Understanding visual semantics embedded in consecutive characters is a crucial capability for both large language models (LLMs) and multi-modal large language models (MLLMs). This type of artifact possesses the unique characteristic that identical information can be readily formulated in both texts and images, making them a significant proxy for analyzing modern LLMs' and MLLMs' capabilities in modality-agnostic vision understanding. In this work, we select ASCII art as a representative artifact, where the lines and brightness used to depict each concept are rendered by characters, and we frame the problem as an ASCII art recognition task. We benchmark model performance on this task by constructing an evaluation dataset with an elaborate categorization tree and also collect a training set to elicit the models' visual perception ability. Through a comprehensive analysis of dozens of models, results reveal that although humans can achieve nearly 100% accuracy, the state-of-the-art LLMs and MLLMs lag far behind. Models are capable of recognizing concepts depicted in the ASCII arts given only text inputs indicated by over 60% accuracy for some concepts, but most of them achieves merely around 30% accuracy when averaged across all categories. When provided with images as inputs, GPT-4o gets 82.68%, outperforming the strongest open-source MLLM by 21.95%. Although models favor different kinds of ASCII art depending on the modality provided, none of the MLLMs successfully benefit when both modalities are supplied simultaneously. Moreover, supervised fine-tuning helps improve models' accuracy especially when provided with the image modality, but also highlights the need for better training techniques to enhance the information fusion among modalities.

Problem

• Benchmarking visual perception ability in text strings using ASCII art
• Evaluating models' recognition of visual semantics embedded in character arrangements
• Assessing multimodal models' performance on ASCII art across text and image inputs

Innovation

• ASCII art benchmark for visual perception evaluation
• Multi-modal analysis of text and image inputs
• Training set for model enhancement provided