Common to Whom? Regional Cultural Commonsense and LLM Bias in India

2026-01-22

AI Summary

This study addresses the limitation of existing cultural commonsense benchmarks, which often treat nations as homogeneous entities and overlook internal regional variation. To remedy this, the authors construct Indica—the first benchmark for evaluating regional cultural commonsense in India—comprising 1,630 human-annotated, region-specific question-answer pairs across eight everyday domains from five major Indian regions. The work systematically assesses large language models’ capacity to understand regional cultural nuances and reveals significant geographic bias: only 39.4% of questions exhibit nationwide consensus, and model accuracy on regional questions ranges merely from 13.4% to 20.9%, with pronounced overrepresentation of north-central perspectives and underestimation of eastern and western regions. The proposed framework, grounded in anthropological principles for question design, regional data collection, and bias quantification, offers a generalizable approach for evaluating cultural competence in other multicultural nations.

Technology Category

• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Natural Language Processing: Question Answering
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Existing cultural commonsense benchmarks treat nations as monolithic, assuming uniform practices within national boundaries. But does cultural commonsense hold uniformly within a nation, or does it vary at the sub-national level? We introduce Indica, the first benchmark designed to test LLMs'ability to address this question, focusing on India - a nation of 28 states, 8 union territories, and 22 official languages. We collect human-annotated answers from five Indian regions (North, South, East, West, and Central) across 515 questions spanning 8 domains of everyday life, yielding 1,630 region-specific question-answer pairs. Strikingly, only 39.4% of questions elicit agreement across all five regions, demonstrating that cultural commonsense in India is predominantly regional, not national. We evaluate eight state-of-the-art LLMs and find two critical gaps: models achieve only 13.4%-20.9% accuracy on region-specific questions, and they exhibit geographic bias, over-selecting Central and North India as the"default"(selected 30-40% more often than expected) while under-representing East and West. Beyond India, our methodology provides a generalizable framework for evaluating cultural commonsense in any culturally heterogeneous nation, from question design grounded in anthropological taxonomy, to regional data collection, to bias measurement.

Problem

• cultural commonsense
• regional variation
• LLM bias
• India
• sub-national diversity

Innovation

• cultural commonsense
• regional bias
• LLM evaluation
• Indica benchmark
• anthropological taxonomy