Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions

2025-01-28

AI Summary

This study systematically evaluates large language models’ (LLMs) understanding of India’s “little traditions”—including caste systems, kinship structures, and localized marriage and religious practices—and their cultural sensitivity, revealing structural limitations wherein generic knowledge dominates and contextual responsiveness falters in indigenous settings. Method: We introduce a novel cross-cultural evaluation paradigm integrating regional-language prompting (e.g., Hindi, Tamil) with analytical frameworks grounded in the “great–little tradition” dichotomy, supported by multi-case studies and a culturally contextualized human evaluation protocol. Contribution/Results: While LLMs demonstrate surface-level generalization of cultural concepts, their situated reasoning remains weak; incorporating native-language prompts significantly improves response relevance, factual accuracy, and cultural appropriateness. This work establishes a reproducible methodological benchmark for assessing LLMs’ cultural competence and provides a structured pathway for diagnosing sociocultural biases in generative AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) have shown remarkable advancements but also raise concerns about cultural bias, often reflecting dominant narratives at the expense of under-represented subcultures. In this study, we evaluate the capacity of LLMs to recognize and accurately respond to the Little Traditions within Indian society, encompassing localized cultural practices and subcultures such as caste, kinship, marriage, and religion. Through a series of case studies, we assess whether LLMs can balance the interplay between dominant Great Traditions and localized Little Traditions. We explore various prompting strategies and further investigate whether using prompts in regional languages enhances the models cultural sensitivity and response quality. Our findings reveal that while LLMs demonstrate an ability to articulate cultural nuances, they often struggle to apply this understanding in practical, context-specific scenarios. To the best of our knowledge, this is the first study to analyze LLMs engagement with Indian subcultures, offering critical insights into the challenges of embedding cultural diversity in AI systems.

Problem

• Large Language Models
• Cultural Understanding
• Local vs General Knowledge

Innovation

• Cultural Competence
• Local Customs Understanding
• Multilingual Language Models