Position: Fairness Failure in Generative Models is an Evaluation Problem

2026-08-17

AI Summary

本文针对生成模型中的公平性问题，提出了一种标准化评估方法——Fairness Cards，旨在提高结果的可比性和可操作性。

Abstract

Despite groundbreaking advancements in generative models during the last decade, concerns about their lack of fairness, reinforcing societal inequalities and harming marginalized groups, remain under-addressed and difficult to act upon. This position paper argues that fairness failures in generative models, albeit driven by multiple factors, are ultimately stemming from an evaluation problem: fairness findings are rarely comparable across papers or actionable for deployment decisions. This paper diagnoses recurring empirical and conceptual failure modes in current practice and motivates a shift from ad-hoc bias checks to standardized, generative-specific evaluation. We propose Fairness Cards as a minimal reporting artifact that makes evaluation choices explicit (prompt families, counterfactual protocols, metrics, and refusal handling) enabling reproducibility, comparability, and accountability. We conclude with additional recommendations towards a paradigm shift in evaluation standards. Our project page can be found at https://mariiavladimirova.github.io/fairness-cards .

Problem

• generative models
• fairness
• societal inequalities
• evaluation problem
• marginalized groups

Innovation

• Fairness Cards
• generative models
• evaluation standards
• reproducibility