EuroGEST: Investigating gender stereotypes in multilingual language models

2025-06-04

AI Summary

To address the longstanding limitation of gender bias evaluation for multilingual large language models (LLMs) being confined to English, this work introduces EuroGEST—the first benchmark covering 30 European languages. Built upon 16 expert-defined stereotype categories, EuroGEST enables reliable cross-lingual extension via translation alignment, morphology-informed heuristics, quality estimation, and human validation. We conduct the first systematic zero-shot gender bias evaluation across 24 multilingual LLMs on 29 non-English European languages. Key findings include: (1) increased model parameter count significantly exacerbates stereotyping; (2) instruction fine-tuning fails to consistently mitigate bias, with effects varying by language and task; and (3) female-associated terms frequently co-occur with “beauty” and “empathy”, whereas male-associated terms strongly correlate with “leadership” and “professionalism”. This work establishes a reproducible benchmark and methodological framework for multilingual fairness research.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models increasingly support multiple languages, yet most benchmarks for gender bias remain English-centric. We introduce EuroGEST, a dataset designed to measure gender-stereotypical reasoning in LLMs across English and 29 European languages. EuroGEST builds on an existing expert-informed benchmark covering 16 gender stereotypes, expanded in this work using translation tools, quality estimation metrics, and morphological heuristics. Human evaluations confirm that our data generation method results in high accuracy of both translations and gender labels across languages. We use EuroGEST to evaluate 24 multilingual language models from six model families, demonstrating that the strongest stereotypes in all models across all languages are that women are extit{beautiful,} extit{empathetic} and extit{neat} and men are extit{leaders}, extit{strong, tough} and extit{professional}. We also show that larger models encode gendered stereotypes more strongly and that instruction finetuning does not consistently reduce gendered stereotypes. Our work highlights the need for more multilingual studies of fairness in LLMs and offers scalable methods and resources to audit gender bias across languages.

Problem

• Measuring gender stereotypes in multilingual language models
• Expanding bias evaluation beyond English to 29 European languages
• Assessing impact of model size and finetuning on stereotype strength

Innovation

• Multilingual dataset for gender bias evaluation
• Translation tools and morphological heuristics
• Human-validated high accuracy translations