Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

2026-07-08

AI Summary

This study addresses the lack of high-quality, low-cost stereotype datasets tailored to the multicultural Spanish-speaking context for evaluating biases in large language models (LLMs). The authors propose a human–AI collaborative annotation framework that leverages LLMs to generate initial stereotype statements, which are then validated by native speakers from multiple European and Latin American countries. Integrating active learning with cost-control strategies, this approach efficiently constructs EspanStereo—the first cross-regional Spanish stereotype dataset. Empirical analysis reveals culturally specific biases absent in English-centric resources and demonstrates significant regional variation in stereotyping behaviors across Spanish-language LLMs. These findings validate the framework’s effectiveness in uncovering localized biases and offer a scalable annotation paradigm applicable to other languages.

Abstract

Research on stereotypes in large language models (LLMs) has largely focused on English-speaking contexts, due to the lack of datasets in other languages and the high cost of manual annotation in underrepresented cultures. To address this gap, we introduce a cost-efficient human-LLM collaborative annotation framework and apply it to construct EspanStereo, a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries across Europe and Latin America. EspanStereo captures both well-documented stereotypes from prior literature and culturally specific biases absent from English-centric resources. Using LLMs to generate candidate stereotypes and in-culture annotators to validate them, we demonstrate the framework's effectiveness in identifying nuanced, region-specific biases. Our evaluation of Spanish-supporting LLMs using EspanStereo reveals significant variation in stereotypical behavior across countries, highlighting the need for more culturally grounded assessments. Beyond Spanish, our framework is adaptable to other languages and regions, offering a scalable path toward multilingual stereotype benchmarks. This work broadens the scope of stereotype analysis in LLMs and lays the groundwork for comprehensive cross-cultural bias evaluation.

Problem

• stereotype
• large language models
• multilingual bias
• cultural specificity
• dataset construction

Innovation

• human-LLM collaboration
• culturally specific stereotypes
• multilingual bias evaluation
• scalable annotation framework
• EspanStereo