The World Wide Recipe: A community-centred framework for fine-grained data collection and regional bias operationalisation

2024-06-13

AI Summary

This work addresses critical limitations of text-to-image (T2I) models in generating region-specific dish imagery—namely, low factual accuracy, cultural misrepresentation, and insufficient cultural sensitivity. We propose the first community-centered, fine-grained recipe data collection framework and release World Wide Dishes, a globally diverse benchmark dataset spanning multiple culinary regions. Methodologically, we introduce a novel three-dimensional cultural bias evaluation framework—assessing accuracy, representativeness, and cultural sensitivity—integrating qualitative community insights with quantitative automated metrics, validated through cross-regional empirical studies (including multi-country African and U.S. comparisons). Our analysis reveals systemic “cultural flattening” and “geographic erasure” in mainstream T2I models: non-Western dishes suffer markedly degraded generation quality, and even within U.S. prompts, stereotypical and distorted outputs frequently occur. We publicly release the dataset and evaluation code to advance fair, culturally grounded generative AI.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

We introduce the World Wide recipe, which sets forth a framework for culturally aware and participatory data collection, and the resultant regionally diverse World Wide Dishes evaluation dataset. We also analyse bias operationalisation to highlight how current systems underperform across several dimensions: (in-)accuracy, (mis-)representation, and cultural (in-)sensitivity, with evidence from qualitative community-based observations and quantitative automated tools. We find that these T2I models generally do not produce quality outputs of dishes specific to various regions. This is true even for the US, which is typically considered more well-resourced in training data -- although the generation of US dishes does outperform that of the investigated African countries. The models demonstrate the propensity to produce inaccurate and culturally misrepresentative, flattening, and insensitive outputs. These representational biases have the potential to further reinforce stereotypes and disproportionately contribute to erasure based on region. The dataset and code are available at https://github.com/oxai/world-wide-dishes.

Problem

• Culturally aware data collection
• Regional bias in models
• Cultural misrepresentation in outputs

Innovation

• Culturally aware data collection
• Regional bias operationalization analysis
• World Wide Dishes dataset creation