From Facts to Folklore: Evaluating Large Language Models on Bengali Cultural Knowledge

2025-10-22

AI Summary

This work addresses the limited cultural knowledge understanding of large language models (LLMs) in low-resource cultural contexts, specifically Bangladeshi culture. To this end, we introduce BLanCK—the first fine-grained Bangla cultural knowledge evaluation dataset—covering dimensions such as folk traditions, culinary culture, and regional dialects. We propose a context-aware prompting framework augmented with culture-specific information injection and conduct systematic evaluations across multilingual LLMs. Results show that while current models perform well on general-purpose tasks, they exhibit significant deficits in culturally grounded reasoning; incorporating culturally contextualized prompts yields substantial performance gains. This demonstrates the critical importance of context sensitivity and culture-aware adaptation for enhancing cultural cognition. Our study establishes a new benchmark and provides a reusable methodology for cultural alignment of LLMs in low-resource settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Language and Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent progress in NLP research has demonstrated remarkable capabilities of large language models (LLMs) across a wide range of tasks. While recent multilingual benchmarks have advanced cultural evaluation for LLMs, critical gaps remain in capturing the nuances of low-resource cultures. Our work addresses these limitations through a Bengali Language Cultural Knowledge (BLanCK) dataset including folk traditions, culinary arts, and regional dialects. Our investigation of several multilingual language models shows that while these models perform well in non-cultural categories, they struggle significantly with cultural knowledge and performance improves substantially across all models when context is provided, emphasizing context-aware architectures and culturally curated training data.

Problem

• Evaluating LLMs on Bengali cultural knowledge gaps
• Addressing limitations in low-resource cultural nuance capture
• Assessing multilingual models' cultural vs non-cultural performance disparities

Innovation

• Developed Bengali cultural knowledge dataset BLanCK
• Evaluated multilingual models' cultural understanding capabilities
• Proposed context-aware architectures for cultural knowledge