AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani

2026-06-13

AI Summary

This study addresses the widespread neglect of subnational cultural diversity in current NLP systems when evaluating sociocultural stereotypical biases, with a particular gap concerning Goa’s multifaceted identity groups. To bridge this gap, the authors introduce AmchiBias, the first bilingual (English and Devanagari-script Konkani) benchmark for assessing stereotypical biases specific to Goa, encompassing eight sociodemographic dimensions and 313 minimal pairs. Evaluation across five multilingual encoders reveals that model performance in Konkani approaches random chance, highlighting severe limitations in low-resource language capabilities. Even when queried in English—despite broader linguistic coverage for pan-Indian groups—the models demonstrate insufficient grasp of Goan local cultural contexts, exposing a critical blind spot in multilingual systems’ capacity for hyperlocal cultural understanding.

Abstract

Socio-cultural stereotypical bias is an important consideration in the development and deployment of NLP systems. It is however often considered only at the national level, despite rich subnational socio-cultural structures. We present AmchiBias, the first benchmark for measuring socio-cultural stereotypical bias for the Indian state of Goa with its unique historically multicultural setting. It covers various Goan identity groups and comprises 313 minimal pairs across eight sociodemographic dimensions in both English and Devanagari Konkani. We then evaluate stereotypical bias in five multilingual encoder models on this benchmark. We find near-chance scores in Konkani, reflecting language incompetence for general multilingual models and a lack of Goan cultural competence for Indian language models. Queried in English, models with a stronger Indian language coverage show higher bias for pan-Indian groups than hyperlocal Goan groups. This suggests the English signal reflects pan-Indian pretraining associations rather than genuine Goan cultural knowledge. Our findings highlight a critical gap in low-resource multilingual NLP evaluation for hyperlocal community identities.

Problem

• socio-cultural stereotypical bias
• subnational identity
• low-resource multilingual NLP
• hyperlocal community
• Goan identity

Innovation

• minimal pair dataset
• socio-cultural bias
• hyperlocal identity
• multilingual NLP
• Konkani language