Aligned but Stereotypical? The Hidden Influence of System Prompts on Social Bias in LVLM-Based Text-to-Image Models

2025-12-04

AI Summary

Large vision-language models (LVLMs) for text-to-image (T2I) generation significantly amplify societal biases, with system prompts acting as key drivers of demographic biases—including gender and racial stereotypes. Method: We propose FairPro, a training-free meta-prompting framework that enables real-time self-auditing and fairness optimization during generation. FairPro operates by decoding intermediate representations, diagnosing token-level probability distributions, and analyzing embedding correlations to identify and mitigate bias propagation. Contribution/Results: We conduct the first systematic evaluation of prompt-induced bias across four levels of linguistic complexity using a 1,024-prompt benchmark. Experiments on SANA and Qwen-Image demonstrate that FairPro substantially reduces social bias while preserving text–image alignment fidelity. Our approach establishes a new paradigm for deployable, lightweight fairness intervention in T2I systems—requiring no model retraining or architectural modification.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large vision-language model (LVLM) based text-to-image (T2I) systems have become the dominant paradigm in image generation, yet whether they amplify social biases remains insufficiently understood. In this paper, we show that LVLM-based models produce markedly more socially biased images than non-LVLM-based models. We introduce a 1,024 prompt benchmark spanning four levels of linguistic complexity and evaluate demographic bias across multiple attributes in a systematic manner. Our analysis identifies system prompts, the predefined instructions guiding LVLMs, as a primary driver of biased behavior. Through decoded intermediate representations, token-probability diagnostics, and embedding-association analyses, we reveal how system prompts encode demographic priors that propagate into image synthesis. To this end, we propose FairPro, a training-free meta-prompting framework that enables LVLMs to self-audit and construct fairness-aware system prompts at test time. Experiments on two LVLM-based T2I models, SANA and Qwen-Image, show that FairPro substantially reduces demographic bias while preserving text-image alignment. We believe our findings provide deeper insight into the central role of system prompts in bias propagation and offer a practical, deployable approach for building more socially responsible T2I systems.

Problem

• LVLM-based T2I models produce socially biased images
• System prompts are a primary driver of demographic bias
• FairPro reduces bias while preserving text-image alignment

Innovation

• Training-free meta-prompting framework for self-auditing
• System prompts analyzed via token-probability and embedding diagnostics
• Constructs fairness-aware prompts to reduce bias in image generation