Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment

2025-09-23

AI Summary

This study reveals that large vision-language models (VLMs) systematically reproduce societal biases—particularly gender–occupation associations—when interpreting news images. Method: We propose the first VLM bias evaluation paradigm integrating fine-grained social attribute annotation with an LLM-as-Judge framework. Our benchmark comprises 1,343 real-world news image–caption pairs and employs a hybrid assessment combining automated classification, LLM-based scoring, and human validation. Contribution/Results: We find that visual context significantly disrupts open-ended generation fidelity, and high output faithfulness does not imply low bias. VLMs exhibit substantial inter-model variation in bias magnitude, confirming the critical role of visual cues in reinforcing social stereotypes. We publicly release the annotated dataset, evaluation protocol, and implementation code, establishing a reproducible methodological foundation for fairness research in VLMs.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large vision-language models (VLMs) can jointly interpret images and text, but they are also prone to absorbing and reproducing harmful social stereotypes when visual cues such as age, gender, race, clothing, or occupation are present. To investigate these risks, we introduce a news-image benchmark consisting of 1,343 image-question pairs drawn from diverse outlets, which we annotated with ground-truth answers and demographic attributes (age, gender, race, occupation, and sports). We evaluate a range of state-of-the-art VLMs and employ a large language model (LLM) as judge, with human verification. Our findings show that: (i) visual context systematically shifts model outputs in open-ended settings; (ii) bias prevalence varies across attributes and models, with particularly high risk for gender and occupation; and (iii) higher faithfulness does not necessarily correspond to lower bias. We release the benchmark prompts, evaluation rubric, and code to support reproducible and fairness-aware multimodal assessment.

Problem

• Evaluating bias risks in vision-language models from social-cue news images
• Assessing how visual context systematically shifts model outputs in open-ended settings
• Investigating bias prevalence across demographic attributes like gender and occupation

Innovation

• News-image benchmark with demographic annotations
• LLM-as-judge assessment with human verification
• Evaluates bias prevalence across attributes and models