Beyond Aesthetics: Cultural Competence in Text-to-Image Models

2024-07-09

arXiv.org

AI Summary

Current text-to-image (T2I) models lack robust cultural knowledge modeling and diversity awareness, and no systematic benchmark exists for evaluating cultural competence. This paper introduces CUBE, the first benchmark dedicated to assessing cultural capability across eight countries, covering food, landmarks, and art—evaluated along two dimensions: cultural awareness and cultural diversity. Methodologically, we propose a structured cultural evaluation framework incorporating a novel quality-weighted Vendi Score to quantify generative diversity. Our contributions include: (1) constructing the first scalable, multi-national cultural artifact benchmark—CUBE-1K and CUBE-CSpace; (2) designing a cultural capability evaluation framework with a quality-aware diversity metric; and (3) developing a structured knowledge base–LLM collaborative data curation pipeline, integrating multi-source verification prompts and semantic alignment assessment. Experiments reveal significant cultural biases in mainstream T2I models and representation imbalance under ambiguous prompts.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Text-to-Image (T2I) models are being increasingly adopted in diverse global communities where they create visual representations of their unique cultures. Current T2I benchmarks primarily focus on faithfulness, aesthetics, and realism of generated images, overlooking the critical dimension of cultural competence. In this work, we introduce a framework to evaluate cultural competence of T2I models along two crucial dimensions: cultural awareness and cultural diversity, and present a scalable approach using a combination of structured knowledge bases and large language models to build a large dataset of cultural artifacts to enable this evaluation. In particular, we apply this approach to build CUBE (CUltural BEnchmark for Text-to-Image models), a first-of-its-kind benchmark to evaluate cultural competence of T2I models. CUBE covers cultural artifacts associated with 8 countries across different geo-cultural regions and along 3 concepts: cuisine, landmarks, and art. CUBE consists of 1) CUBE-1K, a set of high-quality prompts that enable the evaluation of cultural awareness, and 2) CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding to evaluate cultural diversity. We also introduce cultural diversity as a novel T2I evaluation component, leveraging quality-weighted Vendi score. Our evaluations reveal significant gaps in the cultural awareness of existing models across countries and provide valuable insights into the cultural diversity of T2I outputs for under-specified prompts. Our methodology is extendable to other cultural regions and concepts, and can facilitate the development of T2I models that better cater to the global population.

Problem

• Cultural Knowledge
• Diversity Representation
• Evaluation Metrics

Innovation

• Cultural Understanding
• Diversity Generation
• Evaluation Framework