Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

2026-04-21

AI Summary

This work addresses the implicit geographical biases that undermine multilingual large language models when answering regionally ambiguous questions, leading to imbalanced cross-lingual knowledge propagation. The authors introduce LocQA, a benchmark comprising 2,156 questions across 12 languages, to systematically quantify two forms of structural bias: a U.S.-centric global bias at the cross-lingual level and a population-weighted local bias within individual languages. Evaluations across 32 prominent models reveal that instruction tuning exacerbates global bias, highlighting a pervasive reliance on geographical priors. The study provides a principled, quantitative framework and empirical evidence for analyzing and mitigating such biases, offering actionable insights for calibrating model localization behavior.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Multilingual large language models (LLMs) have minimized the fluency gap between languages. This advancement, however, exposes models to the risk of biased behavior, as knowledge and norms may propagate across languages. In this work, we aim to quantify models' inter- and intra-lingual biases, via their ability to answer locale-ambiguous questions. To this end, we present LocQA, a test set containing 2,156 questions in 12 languages, referring to various locale-dependent facts such as laws, dates, and measurements. The questions do not contain indications of the locales they relate to, other than the querying language itself. LLMs' responses to LocQA locale-ambiguous questions thus reveal models' implicit priors. We used LocQA to evaluate 32 models, and detected two types of structural biases. Inter-lingually, we show a global bias towards answers relevant to the US-locale, even when models are asked in languages other than English. Moreover, we discovered that this global bias is exacerbated in models that underwent instruction tuning, compared to their base counterparts. Intra-lingually, we show that when multiple locales are relevant for the same language, models act as demographic probability engines, prioritizing locales with larger populations. Taken together, insights from LocQA may help in shaping LLMs' desired local behavior, and in quantifying the impact of various training phases on different kinds of biases.

Problem

• multilingual LLMs
• locale bias
• inter-lingual bias
• intra-lingual bias
• implicit priors

Innovation

• locale-ambiguous questions
• multilingual LLMs
• implicit bias
• LocQA
• instruction tuning