Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES)

2025-10-28

AI Summary

Existing NLP bias evaluation benchmarks are heavily Anglo-centric and U.S.-focused, resulting in inadequate cultural representation—particularly for regions such as Latin America—and limiting the identification and mitigation of locally embedded stereotypes. To address this, we propose the first dynamic, Latin America–specific social bias evaluation framework, built upon community-driven qualitative research, a structured annotation protocol, and iterative validation—enabling synergistic item acquisition and quality control of existing data. Leveraging this framework, we construct LACES, a large-scale, region-sensitive dataset featuring multiple Spanish variants across Latin American countries and fine-grained social context annotations. LACES significantly enhances modeling and detection of regionally salient stereotypes. Our work fills a critical geographical and cultural gap in fairness evaluation and introduces a transferable methodological paradigm—“community participation, dynamic iteration, cross-contextual validation”—advancing NLP bias research from universalist to contextually grounded approaches.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The evaluation of societal biases in NLP models is critically hindered by a glaring geo-cultural gap, as existing benchmarks are overwhelmingly English-centric and focused on U.S. demographics. This leaves regions such as Latin America severely underserved, making it impossible to adequately assess or mitigate the perpetuation of harmful regional stereotypes by language technologies. To address this gap, we introduce a new, large-scale dataset of stereotypes developed through targeted community partnerships within Latin America. Furthermore, we present a novel dynamic data collection methodology that uniquely integrates the sourcing of new stereotype entries and the validation of existing data within a single, unified workflow. This combined approach results in a resource with significantly broader coverage and higher regional nuance than static collection methods. We believe that this new method could be applicable in gathering sociocultural knowledge of other kinds, and that this dataset provides a crucial new resource enabling robust stereotype evaluation and significantly addressing the geo-cultural deficit in fairness resources for Latin America.

Problem

• Addressing geo-cultural bias in NLP model evaluation
• Developing Latin American stereotype dataset via community partnerships
• Creating dynamic data collection for regional stereotype validation

Innovation

• Dynamic data collection integrating sourcing and validation
• Community-sourced stereotypes dataset for Latin America
• Unified workflow for broader regional nuance coverage