Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

2026-04-17

AI Summary

This study investigates whether vision-language models genuinely rely on visual information for reasoning. To this end, we introduce CrossMath, the first multimodal mathematical reasoning benchmark enabling rigorous cross-modal comparison, with human-verified alignment among text-only, image-only, and combined text-image questions. Systematic evaluation reveals that current models achieve their best performance on text-only inputs, while the inclusion of images often degrades accuracy—highlighting a significant deficiency in their visual reasoning capabilities. Fine-tuning on CrossMath substantially enhances model performance across all modalities and generalizes to broader visual reasoning tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.

Problem

• vision-language models
• vision reasoning
• modality gap
• multimodal reasoning
• visual evidence

Innovation

• vision-language models
• modality gap
• CrossMath
• visual reasoning
• multimodal benchmark