Evaluating Perspectival Biases in Cross-Modal Retrieval

2025-10-30

AI Summary

This work identifies two novel types of perspective bias in cross-modal retrieval: *linguistic popularity bias*—where image-to-text retrieval favors high-frequency linguistic items over semantically optimal ones—and *cultural associativity bias*—where text-to-image retrieval prefers culturally stereotypical images over semantically accurate ones. We propose a multilingual and multicultural empirical framework grounded in cross-modal alignment analysis, systematically evaluating bias origins and mitigation strategies across a dataset spanning 12 languages and six cultural regions. Our findings show that explicit alignment significantly mitigates popularity bias but yields limited improvement for cultural associativity bias, indicating its entrenchment in deep semantic representations and necessitating structured interventions beyond data augmentation. This study is the first to formally define, distinguish, and comparatively analyze these two biases, establishing cultural associativity bias as more persistent and challenging. The work provides both theoretical foundations and methodological paradigms for developing fairer cross-modal retrieval systems.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal retrieval systems are expected to operate in a semantic space, agnostic to the language or cultural origin of the query. In practice, however, retrieval outcomes systematically reflect perspectival biases: deviations shaped by linguistic prevalence and cultural associations. We study two such biases. First, prevalence bias refers to the tendency to favor entries from prevalent languages over semantically faithful entries in image-to-text retrieval. Second, association bias refers to the tendency to favor images culturally associated with the query over semantically correct ones in text-to-image retrieval. Results show that explicit alignment is a more effective strategy for mitigating prevalence bias. However, association bias remains a distinct and more challenging problem. These findings suggest that achieving truly equitable multimodal systems requires targeted strategies beyond simple data scaling and that bias arising from cultural association may be treated as a more challenging problem than one arising from linguistic prevalence.

Problem

• Evaluating perspectival biases in cross-modal retrieval systems
• Analyzing prevalence bias favoring dominant languages in retrieval
• Examining association bias prioritizing culturally linked images

Innovation

• Explicit alignment mitigates prevalence bias effectively
• Association bias remains a distinct challenging problem
• Targeted strategies needed beyond simple data scaling