AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment

2025-09-15

AI Summary

This paper systematically investigates identity bias in multimodal large language models (MLLMs) for personalized image aesthetic assessment and evaluates their alignment with human aesthetic preferences. To this end, we introduce AesBiasBench—the first benchmark dedicated to aesthetic bias evaluation—comprising three subtasks: aesthetic perception, aesthetic assessment, and aesthetic empathy. It is the first framework to jointly model stereotypical bias and human preference alignment. We propose structured metrics—including Identity Bias Strength (IFD), Normalized Response Difference (NRD), and Alignment Accuracy Score (AAS)—and integrate demographic subgroup analysis with ground-truth human preference data to quantify biases across gender, age, and education levels. Experimental evaluation across 19 state-of-the-art MLLMs reveals that smaller models exhibit significantly stronger identity bias; while larger models generally align better with human preferences, explicitly injecting identity information exacerbates emotional judgment bias.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Multimodal Large Language Models (MLLMs) are increasingly applied in Personalized Image Aesthetic Assessment (PIAA) as a scalable alternative to expert evaluations. However, their predictions may reflect subtle biases influenced by demographic factors such as gender, age, and education. In this work, we propose AesBiasBench, a benchmark designed to evaluate MLLMs along two complementary dimensions: (1) stereotype bias, quantified by measuring variations in aesthetic evaluations across demographic groups; and (2) alignment between model outputs and genuine human aesthetic preferences. Our benchmark covers three subtasks (Aesthetic Perception, Assessment, Empathy) and introduces structured metrics (IFD, NRD, AAS) to assess both bias and alignment. We evaluate 19 MLLMs, including proprietary models (e.g., GPT-4o, Claude-3.5-Sonnet) and open-source models (e.g., InternVL-2.5, Qwen2.5-VL). Results indicate that smaller models exhibit stronger stereotype biases, whereas larger models align more closely with human preferences. Incorporating identity information often exacerbates bias, particularly in emotional judgments. These findings underscore the importance of identity-aware evaluation frameworks in subjective vision-language tasks.

Problem

• Evaluating bias in multimodal models for aesthetic assessment
• Assessing alignment between model outputs and human preferences
• Measuring demographic stereotype biases in aesthetic evaluations

Innovation

• AesBiasBench benchmark for bias evaluation
• Structured metrics to assess stereotype bias
• Identity-aware framework for multimodal models