KAHANI: Culturally-Nuanced Visual Storytelling Pipeline for Non-Western Cultures

2024-10-25

arXiv.org

AI Summary

Contemporary AIGC models—including large language models (LLMs) and text-to-image (T2I) systems—are predominantly trained on and aligned with Global North cultural paradigms, leading to “othering” gaze toward non-Western cultures and undermining the authenticity of local narratives. Method: We propose KAHANI—the first lightweight, culturally grounded visual storytelling pipeline for non-Western contexts—integrating Chain-of-Thought reasoning with culture-aware hierarchical prompting to explicitly model cultural context and generate Culture-Specific Items (CSIs). Built upon GPT-4 Turbo and Stable Diffusion XL, it prioritizes cultural fidelity without sacrificing efficiency. Contribution/Results: Evaluated via a cross-regional user study in India, KAHANI demonstrates strong cultural adaptability. Across 36 quantitative and qualitative metrics, it significantly outperforms ChatGPT-4 + DALL·E 3 on 27, markedly enhancing cultural relevance, narrative coherence, and visual fidelity in culturally situated storytelling.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Large Language Models (LLMs) and Text-To-Image (T2I) models have demonstrated the ability to generate compelling text and visual stories. However, their outputs are predominantly aligned with the sensibilities of the Global North, often resulting in an outsider's gaze on other cultures. As a result, non-Western communities have to put extra effort into generating culturally specific stories. To address this challenge, we developed a visual storytelling pipeline called KAHANI that generates culturally grounded visual stories for non-Western cultures. Our pipeline leverages off-the-shelf models GPT-4 Turbo and Stable Diffusion XL (SDXL). By using Chain of Thought (CoT) and T2I prompting techniques, we capture the cultural context from user's prompt and generate vivid descriptions of the characters and scene compositions. To evaluate the effectiveness of KAHANI, we conducted a comparative user study with ChatGPT-4 (with DALL-E3) in which participants from different regions of India compared the cultural relevance of stories generated by the two tools. Results from the qualitative and quantitative analysis performed on the user study showed that KAHANI was able to capture and incorporate more Culturally Specific Items (CSIs) compared to ChatGPT-4. In terms of both its cultural competence and visual story generation quality, our pipeline outperformed ChatGPT-4 in 27 out of the 36 comparisons.

Problem

• Addresses lack of cultural nuance in LLM and T2I outputs for non-Western cultures.
• Develops Kahani to generate culturally grounded visual stories for non-Western communities.
• Evaluates Kahani's effectiveness in capturing cultural nuances compared to ChatGPT-4.

Innovation

• Leverages GPT-4 Turbo and Stable Diffusion XL
• Uses Chain of Thought and T2I prompting
• Generates culturally nuanced visual stories