Evaluation of Geographical Distortions in Language Models: A Crucial Step Towards Equitable Representations

2024-04-26

IFIP Working Conference on Database Semantics

AI Summary

This paper identifies a pervasive geographic representation distortion in large language models (LLMs), wherein semantic distances between locations systematically deviate from true geospatial distances—particularly overestimating distances involving underdeveloped regions. To address this, we propose, for the first time, four geographic–semantic distance contrast metrics that jointly leverage coordinate embeddings, cosine similarity, geodesic distance computation, and cross-model consistency analysis, establishing a novel evaluation paradigm grounded in spatial fairness. Empirical evaluation across ten state-of-the-art LLMs reveals statistically significant geographic bias in all models, confirming both the universality and structural nature of spatial distortion. Our work provides a quantifiable, reproducible methodological foundation for geographic representation calibration, spatial fairness assurance, and enhanced LLM trustworthiness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Language models now constitute essential tools for improving efficiency for many professional tasks such as writing, coding, or learning. For this reason, it is imperative to identify inherent biases. In the field of Natural Language Processing, five sources of bias are well-identified: data, annotation, representation, models, and research design. This study focuses on biases related to geographical knowledge. We explore the connection between geography and language models by highlighting their tendency to misrepresent spatial information, thus leading to distortions in the representation of geographical distances. This study introduces four indicators to assess these distortions, by comparing geographical and semantic distances. Experiments are conducted from these four indicators with ten widely used language models. Results underscore the critical necessity of inspecting and rectifying spatial biases in language models to ensure accurate and equitable representations.

Problem

• Identifying geographical biases in language models
• Measuring spatial distortions in semantic representations
• Evaluating distance misrepresentations across multiple models

Innovation

• Introduces four indicators for geographical distortion evaluation
• Compares geographical and semantic distances in language models
• Tests ten widely used models to assess spatial biases