SESGO: Spanish Evaluation of Stereotypical Generative Outputs

2025-09-03

AI Summary

Existing multilingual large language models (LLMs) lack rigorous, culturally grounded evaluation of social biases—particularly in Spanish and Latin American contexts. Method: We introduce the first culturally anchored Spanish-language bias benchmark, covering four sensitive dimensions—gender, race, socioeconomic status, and nationality—using 4,000+ regionally adapted prompts, including proverbs and underspecified queries. We evaluate leading commercial LLMs and propose a novel metric integrating answer accuracy and bias directionality. Contribution/Results: We find that English-centric debiasing techniques exhibit limited efficacy in Spanish, and bias patterns remain consistent across temperature settings. Our work delivers a reproducible, multilingual fairness evaluation benchmark and a modular assessment architecture, empirically validating the necessity of cross-lingual bias evaluation and advancing research on multilingual AI fairness.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper addresses the critical gap in evaluating bias in multilingual Large Language Models (LLMs), with a specific focus on Spanish language within culturally-aware Latin American contexts. Despite widespread global deployment, current evaluations remain predominantly US-English-centric, leaving potential harms in other linguistic and cultural contexts largely underexamined. We introduce a novel, culturally-grounded framework for detecting social biases in instruction-tuned LLMs. Our approach adapts the underspecified question methodology from the BBQ dataset by incorporating culturally-specific expressions and sayings that encode regional stereotypes across four social categories: gender, race, socioeconomic class, and national origin. Using more than 4,000 prompts, we propose a new metric that combines accuracy with the direction of error to effectively balance model performance and bias alignment in both ambiguous and disambiguated contexts. To our knowledge, our work presents the first systematic evaluation examining how leading commercial LLMs respond to culturally specific bias in the Spanish language, revealing varying patterns of bias manifestation across state-of-the-art models. We also contribute evidence that bias mitigation techniques optimized for English do not effectively transfer to Spanish tasks, and that bias patterns remain largely consistent across different sampling temperatures. Our modular framework offers a natural extension to new stereotypes, bias categories, or languages and cultural contexts, representing a significant step toward more equitable and culturally-aware evaluation of AI systems in the diverse linguistic environments where they operate.

Problem

• Evaluating bias in multilingual LLMs for Spanish language
• Addressing culturally-aware Latin American stereotype detection
• Assessing bias mitigation transfer from English to Spanish contexts

Innovation

• Culturally-grounded framework for bias detection
• Adapts underspecified questions with cultural expressions
• New metric combining accuracy and error direction