StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs

2026-06-18

AI Summary

This work addresses the unclear role of visual cues in the social biases exhibited by multimodal large language models (MLLMs) in high-stakes societal contexts. The authors introduce StylisticBias, a novel benchmark that systematically disentangles and evaluates the influence of individual visual attributes on 25 binary social judgments, while holding facial identity constant. By generating 500 base faces and approximately 25K single-attribute variants, the study reveals that age and body size dominate identity-level effects, whereas a few attributes—such as fashion style—induce the largest judgment shifts. Notably, around 15 attributes account for nearly 80% of bias variance, highlighting the high sparsity and strong attributability of social biases in MLLMs.

Abstract

Multimodal large language models (MLLMs) are increasingly deployed in personally and societally consequential settings, yet the visual cues that shape how these models judge people remain poorly understood. Prior work often compares different (groups of) individuals, making it difficult to separate appearance effects from identity differences. We introduce StylisticBias, a controlled benchmark for evaluating attribute-level social bias in MLLMs. We generate 500 photorealistic base faces and create about 50 single-attribute variations per face, producing about 25K images. This design keeps identity fixed and changes one visual attribute at a time. It lets us measure how specific cues shift model judgments. We evaluate six MLLMs across 25 binary social judgment scenarios. We find that age and body type dominate identity-level effects, while fashion style and other visual cues drive the largest attribute-level shifts. We further find that about 15 attributes account for nearly 80\% of the total variation, showing that bias is concentrated in a small set of visual cues. Sensitivity is strongest in judgments that are semantically aligned with appearance, especially socioeconomic and style-related judgments. We release StylisticBias as a benchmark for fine-grained bias evaluation in multimodal models. Code and dataset: https://github.com/timo-cavelius/StylisticBias and https://hf.co/datasets/shaghayegh/stylistic-bias-dataset.

Problem

• social bias
• visual cues
• multimodal large language models
• appearance effects
• attribute-level bias

Innovation

• StylisticBias
• multimodal large language models
• attribute-level bias
• controlled visual benchmark
• social bias evaluation