LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?

2025-05-18

AI Summary

This work addresses the lack of dedicated evaluation benchmarks for logical reasoning over text-dense images in large multimodal models (LMMs). We introduce LogicOCR, the first benchmark explicitly designed for logical reasoning on such images: it comprises 1,100 multiple-choice questions derived from Chinese public examination corpora, generated via a GPT-4V-driven automated image-text synthesis pipeline and rigorously validated by human experts to ensure high fidelity and diversity. Our evaluation paradigm is weakly domain-dependent but strongly logic-oriented, supported by a scalable synthesis pipeline covering layout, typography, and visual realism. We conduct systematic evaluations of both open- and closed-source LMMs using chain-of-thought prompting, test-time scaling, and multidimensional attribution analysis. Results reveal a substantial performance gap between text-dense image reasoning and pure textual reasoning; further, we identify critical bottlenecks—including sensitivity to modality input format and image-text orientation—as key limitations hindering current LMM capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in Large Multimodal Models (LMMs) have significantly improved their reasoning and Optical Character Recognition (OCR) capabilities. However, their performance on complex logical reasoning tasks involving text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 1,100 multiple-choice questions designed to evaluate LMMs' logical reasoning abilities on text-rich images, while minimizing reliance on domain-specific knowledge (e.g., mathematics). We construct LogicOCR by curating a text corpus from the Chinese National Civil Servant Examination and develop a scalable, automated pipeline to convert it into multimodal samples. First, we design prompt templates to steer GPT-Image-1 to generate images with diverse backgrounds, interleaved text-illustration layouts, and varied fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified, with low-quality examples discarded. We evaluate a range of representative open-source and proprietary LMMs under both Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. We hope LogicOCR will serve as a valuable resource for advancing multimodal reasoning research. The dataset is available at https://github.com/MiliLab/LogicOCR.

Problem

• Evaluate LMMs' logical reasoning on text-rich images
• Assess performance gap in multimodal versus text-only reasoning
• Develop benchmark for visual-text reading and reasoning integration

Innovation

• Automated pipeline for multimodal sample conversion
• GPT-Image-1 generates diverse text-rich images
• Benchmark evaluates LMMs' logical reasoning abilities