You Prefer This One, I Prefer Yours: Using Reference Words is Harder Than Vocabulary Words for Humans and Multimodal Language Models

2025-05-29

AI Summary

This study investigates cognitive limitations of multimodal language models (MLMs) in referential comprehension—specifically, possessive pronouns (e.g., “my”/“your”) and demonstratives (e.g., “this”/“that”)—which are frequent yet challenging in everyday communication. Method: We construct the first cognitively grounded hierarchy of referential difficulty, and systematically evaluate seven state-of-the-art MLMs via human behavioral benchmarks, cross-modal coreference resolution tasks, and prompt-engineering interventions. Contribution/Results: MLMs significantly underperform humans on demonstrative comprehension; prompt engineering yields only marginal gains for possessives and fails to close the demonstrative gap. We identify a fundamental structural deficit: insufficient perspective-taking and pragmatic inference capabilities. This work provides the first empirical characterization of MLMs’ critical shortcomings in social cognition, establishing a new evaluation benchmark and theoretical foundation for advancing multimodal language understanding.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Multimodal language models (MLMs) increasingly communicate in human-like ways, yet their ability to use reference words remains largely overlooked despite their ubiquity in everyday communication. Our study addresses this gap by comparing human and MLM use of three word classes with increasing cognitive demands: vocabulary words, possessive pronouns (`mine' vs `yours'), and demonstrative pronouns (`this one' vs `that one'). Evaluating seven state-of-the-art MLMs against human participants, we observe a clear difficulty hierarchy: while MLMs approach human-level performance on the vocabulary task, they show substantial deficits with possessives and demonstratives. Our analysis reveals these difficulties stem from limitations in perspective-taking and spatial reasoning. Although prompt engineering improved model performance on possessive use, demonstrative use remained well below human-level competence. These findings provide theoretical and empirical evidence that producing grammatical forms requiring pragmatics and social cognition remains a clear challenge in current NLP systems.

Problem

• Evaluating MLMs' difficulty in using reference words compared to vocabulary
• Assessing deficits in possessives and demonstratives due to perspective-taking limits
• Identifying challenges in grammatical forms requiring pragmatics and social cognition

Innovation

• Comparing human and MLM reference word usage
• Identifying MLM deficits in perspective-taking
• Improving possessive use via prompt engineering