Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?

2025-03-13

AI Summary

This study investigates divergent reliance on world knowledge versus linguistic bias between large language models (LLMs) and humans in syntactic ambiguity resolution, focusing on relative clause attachment ambiguity across six typologically diverse languages (English, Chinese, Japanese, Korean, Russian, Spanish). We introduce MultiWho—the first cross-lingual, fine-grained dataset for relative clause attachment preference—constructed via multilingual prompt engineering, expert human annotation, and a preference modeling–based evaluation framework, benchmarked against controlled human behavioral experiments. Results show that while LLMs perform robustly on unambiguous sentences, they exhibit rigid dependence on world knowledge and a pervasive bias toward local attachment in ambiguous cases, failing to accommodate grammatical variation or language-specific syntactic constraints. Critically, this work provides the first systematic evidence that LLMs lack the grammar–semantics co-processing capability observed in humans, revealing a fundamental limitation in their cross-lingual syntactic reasoning capacity.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study explores how recent large language models (LLMs) navigate relative clause attachment {ambiguity} and use world knowledge biases for disambiguation in six typologically diverse languages: English, Chinese, Japanese, Korean, Russian, and Spanish. We describe the process of creating a novel dataset -- MultiWho -- for fine-grained evaluation of relative clause attachment preferences in ambiguous and unambiguous contexts. Our experiments with three LLMs indicate that, contrary to humans, LLMs consistently exhibit a preference for local attachment, displaying limited responsiveness to syntactic variations or language-specific attachment patterns. Although LLMs performed well in unambiguous cases, they rigidly prioritized world knowledge biases, lacking the flexibility of human language processing. These findings highlight the need for more diverse, pragmatically nuanced multilingual training to improve LLMs' handling of complex structures and human-like comprehension.

Problem

• Explores LLMs' handling of syntactic ambiguity in diverse languages.
• Compares LLMs' and humans' reliance on world knowledge for disambiguation.
• Highlights LLMs' limitations in flexible, human-like language processing.

Innovation

• Created MultiWho dataset for evaluation.
• Tested LLMs on six diverse languages.
• Highlighted LLMs' local attachment preference.