Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

2026-03-19

AI Summary

This work addresses a critical gap in multimodal large language models (MLLMs): their mismatch between perceptual and reasoning capabilities when interpreting discrete symbolic notations—such as mathematical expressions or chemical structures—hindering genuine mastery of the symbolic languages underpinning scientific discovery. To investigate this, the authors construct a comprehensive benchmark spanning five domains: language, culture, mathematics, physics, and chemistry. Their analysis reveals a previously unreported “cognitive mismatch” phenomenon, wherein models often fail at basic symbol recognition yet succeed in complex reasoning tasks, exposing a fundamental reliance on linguistic priors rather than authentic visual perception. Through extensive cross-domain evaluation and qualitative–quantitative diagnostics, the study systematically identifies key weaknesses in current MLLMs’ symbolic understanding and offers crucial guidance for developing more rigorous, human-aligned multimodal intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While Multimodal Large Language Models (MLLMs) have achieved remarkable success in interpreting natural scenes, their ability to process discrete symbols -- the fundamental building blocks of human cognition -- remains a critical open question. Unlike continuous visual data, symbols such as mathematical formulas, chemical structures, and linguistic characters require precise, deeper interpretation. This paper introduces a comprehensive benchmark to evaluate how top-tier MLLMs navigate these "discrete semantic spaces" across five domains: language, culture, mathematics, physics, and chemistry. Our investigation uncovers a counterintuitive phenomenon: models often fail at basic symbol recognition yet succeed in complex reasoning tasks, suggesting they rely on linguistic probability rather than true visual perception. By exposing this "cognitive mismatch", we highlight a significant gap in current AI capabilities: the struggle to truly perceive and understand the symbolic languages that underpin scientific discovery and abstract thought. This work offers a roadmap for developing more rigorous, human-aligned intelligent systems.

Problem

• Cognitive Mismatch
• Multimodal Large Language Models
• Discrete Symbols
• Symbol Understanding
• Symbolic Reasoning

Innovation

• cognitive mismatch
• multimodal large language models
• discrete symbols
• symbolic understanding
• comprehensive benchmark