RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation

2025-02-11

AI Summary

Text-to-image generation models exhibit pervasive English cultural bias, leading to misrepresentation and stereotyping of non-English cultures—such as Russian culture—and thereby introducing risks of bias and offense. To address this, we introduce RusCode, the first benchmark for evaluating visual representation of Russian culture. It systematically defines 19 Russian cultural visual codes and comprises a bilingual (Russian–English) prompt set of 1,250 instances. Our methodology integrates culturally sensitive prompt engineering, a human-annotated taxonomy, and multidimensional human evaluation—assessing faithfulness, recognizability, and cultural appropriateness. Experiments reveal substantial inaccuracies in mainstream models’ generation of Russian cultural concepts. This work fills a critical gap in cross-cultural generative evaluation within computer vision, establishing a reproducible diagnostic baseline and a culturally adaptive evaluation paradigm.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Text-to-image generation models have gained popularity among users around the world. However, many of these models exhibit a strong bias toward English-speaking cultures, ignoring or misrepresenting the unique characteristics of other language groups, countries, and nationalities. The lack of cultural awareness can reduce the generation quality and lead to undesirable consequences such as unintentional insult, and the spread of prejudice. In contrast to the field of natural language processing, cultural awareness in computer vision has not been explored as extensively. In this paper, we strive to reduce this gap. We propose a RusCode benchmark for evaluating the quality of text-to-image generation containing elements of the Russian cultural code. To do this, we form a list of 19 categories that best represent the features of Russian visual culture. Our final dataset consists of 1250 text prompts in Russian and their translations into English. The prompts cover a wide range of topics, including complex concepts from art, popular culture, folk traditions, famous people's names, natural objects, scientific achievements, etc. We present the results of a human evaluation of the side-by-side comparison of Russian visual concepts representations using popular generative models.

Problem

• Bias in text-to-image models
• Lack of Russian cultural representation
• Need for cultural awareness benchmarks

Innovation

• RusCode benchmark creation
• Cultural code integration
• Human evaluation implementation