From Global to Local: Social Bias Transfer in CLIP

2025-08-25

AI Summary

This study investigates the mechanism by which social biases in contrastive language–image pretraining (CLIP) models transfer from pretraining to downstream multimodal tasks—such as visual question answering and image captioning. Method: Through empirical analysis across multiple CLIP variants, downstream tasks, and diverse data subsets, we quantify bias at both pretraining and task-specific stages, examining representational dynamics during adaptation. Contribution/Results: We identify significant variance in bias measurements across subsets and find no stable correlation between pretraining bias and downstream bias manifestation. Crucially, we observe representational convergence during downstream adaptation, rendering bias propagation paths unpredictable. We formally introduce the phenomenon of *bias transfer inconsistency*, challenging the implicit assumption of linear bias transmission. This finding offers a novel perspective on bias evolution in multimodal foundation models and exposes critical limitations in current evaluation paradigms for modeling bias transferability.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Multimodal Learning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The recycling of contrastive language-image pre-trained (CLIP) models as backbones for a large number of downstream tasks calls for a thorough analysis of their transferability implications, especially their well-documented reproduction of social biases and human stereotypes. How do such biases, learned during pre-training, propagate to downstream applications like visual question answering or image captioning? Do they transfer at all? We investigate this phenomenon, referred to as bias transfer in prior literature, through a comprehensive empirical analysis. Firstly, we examine how pre-training bias varies between global and local views of data, finding that bias measurement is highly dependent on the subset of data on which it is computed. Secondly, we analyze correlations between biases in the pre-trained models and the downstream tasks across varying levels of pre-training bias, finding difficulty in discovering consistent trends in bias transfer. Finally, we explore why this inconsistency occurs, showing that under the current paradigm, representation spaces of different pre-trained CLIPs tend to converge when adapted for downstream tasks. We hope this work offers valuable insights into bias behavior and informs future research to promote better bias mitigation practices.

Problem

• Analyzing social bias transfer from CLIP pre-training to downstream tasks
• Investigating bias propagation in visual question answering and image captioning
• Examining inconsistent bias trends between pre-trained models and applications

Innovation

• Analyzing bias transfer in CLIP models
• Comparing global versus local data views
• Examining representation space convergence downstream