Culture Cartography: Mapping the Landscape of Cultural Knowledge

2025-10-31

AI Summary

Large language models (LLMs) lack indigenous cultural knowledge due to its underrepresentation in pretraining corpora. Method: We propose CultureCartography—a human-in-the-loop, bidirectional knowledge mining framework that identifies low-confidence cultural QA pairs and engages users to actively supplement, correct, and annotate them. It integrates confidence-driven active sampling, an interactive annotation interface, and CultureExplorer—a dynamic knowledge exploration system—to enable efficient, interpretable cultural knowledge acquisition. Contribution/Results: Evaluated with real users, CultureCartography constructs a high-quality cultural knowledge dataset, boosting Llama-3.1-8B’s accuracy on cultural benchmarks by up to +19.2%, outperforming existing open-source models and web-search baselines. Its core contribution is the first “user-led, model-targeted” hybrid active learning paradigm, offering a scalable and verifiable pathway for cultural alignment of LLMs.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

To serve global users safely and productively, LLMs need culture-specific knowledge that might not be learned during pre-training. How do we find such knowledge that is (1) salient to in-group users, but (2) unknown to LLMs? The most common solutions are single-initiative: either researchers define challenging questions that users passively answer (traditional annotation), or users actively produce data that researchers structure as benchmarks (knowledge extraction). The process would benefit from mixed-initiative collaboration, where users guide the process to meaningfully reflect their cultures, and LLMs steer the process towards more challenging questions that meet the researcher's goals. We propose a mixed-initiative methodology called CultureCartography. Here, an LLM initializes annotation with questions for which it has low-confidence answers, making explicit both its prior knowledge and the gaps therein. This allows a human respondent to fill these gaps and steer the model towards salient topics through direct edits. We implement this methodology as a tool called CultureExplorer. Compared to a baseline where humans answer LLM-proposed questions, we find that CultureExplorer more effectively produces knowledge that leading models like DeepSeek R1 and GPT-4o are missing, even with web search. Fine-tuning on this data boosts the accuracy of Llama-3.1-8B by up to 19.2% on related culture benchmarks.

Problem

• Mapping cultural knowledge gaps in LLMs
• Developing mixed-initiative human-LLM collaboration
• Improving LLM cultural accuracy through targeted fine-tuning

Innovation

• Mixed-initiative collaboration between users and LLMs
• LLM initializes annotation with low-confidence questions
• Human respondents fill gaps and steer model topics