Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

2025-12-09

AI Summary

This work identifies a critical cross-modal inconsistency problem in multimodal large language models (MLLMs): inconsistent reasoning outputs across image-only, text-only, and multimodal (image-text) inputs for semantically identical content. To address this, we introduce REST/REST+, the first rendering-equivalent stress-testing benchmark, and propose a cross-modal consistency evaluation framework integrating rendering-equivalent sample generation, OCR-robustness control, and visual-token-to-image-attribute attribution analysis. Empirical evaluation across 15 state-of-the-art MLLMs reveals: (1) visual features—particularly text color and image resolution—and visual token count are primary determinants of consistency; (2) OCR accuracy alone does not guarantee consistent reasoning; and (3) consistency scores strongly correlate with the image-text modality gap, enabling mechanistic interpretation of semantic alignment bottlenecks. This work establishes a novel paradigm for trustworthy MLLM evaluation and alignment optimization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce two new benchmarks REST and REST+(Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs). MLLMs are trained to represent vision and language in the same embedding space, yet they cannot perform the same tasks in both modalities. Our benchmarks contain samples with the same semantic information in three modalities (image, text, mixed) and we show that state-of-the-art MLLMs cannot consistently reason over these different modalities. We evaluate 15 MLLMs and find that the degree of modality inconsistency varies substantially, even when accounting for problems with text recognition (OCR). Neither rendering text as image nor rendering an image as text solves the inconsistency. Even if OCR is correct, we find that visual characteristics (text colour and resolution, but not font) and the number of vision tokens have an impact on model performance. Finally, we find that our consistency score correlates with the modality gap between text and images, highlighting a mechanistic interpretation of cross-modal inconsistent MLLMs.

Problem

• Evaluates cross-modal inconsistency in MLLMs
• Assesses reasoning consistency across image, text, mixed modalities
• Investigates impact of visual characteristics on model performance

Innovation

• Benchmarks REST and REST+ evaluate cross-modal inconsistency
• Tests same semantic information across image, text, mixed modalities
• Analyzes visual characteristics and vision tokens impact on performance