Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation

2026-02-04

AI Summary

This study investigates why multilingual language models consistently underperform monolingual models on word sense disambiguation tasks, a phenomenon whose underlying causes remain poorly understood. The authors construct a controlled English–Spanish bilingual dataset for ambiguous word relatedness judgment and systematically compare the performance of monolingual and multilingual models. For the first time, they attribute the performance gap to three capacity-related constraints: representation anisotropy, attention dispersion, and multi-token subword segmentation. Through quantitative analyses—including embedding isotropy measurements, attention weight inspection, and subword token statistics—they demonstrate that multilingual models exhibit pronounced weaknesses across all three dimensions, collectively accounting for their diminished disambiguation capability. This work offers a novel perspective on the intrinsic limitations of multilingual models in lexical semantic tasks.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multilingual language models (LMs) sometimes under-perform their monolingual counterparts, possibly due to capacity limitations. We quantify this ``multilingual penalty''for lexical disambiguation--a task requiring precise semantic representations and contextualization mechanisms--using controlled datasets of human relatedness judgments for ambiguous words in both English and Spanish. Comparing monolingual and multilingual LMs from the same families, we find consistently reduced performance in multilingual LMs. We then explore three potential capacity constraints: representational (reduced embedding isotropy), attentional (reduced attention to disambiguating cues), and vocabulary-related (increased multi-token segmentation). Multilingual LMs show some evidence of all three limitations; moreover, these factors statistically account for the variance formerly attributed to a model's multilingual status. These findings suggest both that multilingual LMs do suffer from multiple capacity constraints, and that these constraints correlate with reduced disambiguation performance.

Problem

• multilingual penalty
• lexical disambiguation
• capacity constraints
• language models
• semantic representation

Innovation

• multilingual penalty
• lexical disambiguation
• capacity constraints
• embedding isotropy
• attention mechanisms