Datasets for Fairness in Language Models: An In-Depth Survey

2025-06-29

AI Summary

Current fairness evaluations of language models often neglect systematic auditing of benchmark datasets themselves, leading to unclear implicit assumptions, demographic coverage biases, and ill-defined applicability boundaries—thereby undermining the reliability of fairness assessments. Method: We conduct the first multi-dimensional audit of 24 prominent fairness benchmark datasets across provenance, scope, content, and usage, and propose a unified evaluation framework. Leveraging quantitative analysis, we identify persistent demographic disparity patterns across datasets and scoring methodologies. Contribution/Results: We innovatively treat datasets—not just model outputs—as primary sources of bias. We release open-source code, structured metadata, and practical guidelines to promote principled dataset selection, transparent evaluation, and the development of socially diverse, equity-oriented benchmarks.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Fairness benchmarks play a central role in shaping how we evaluate language models, yet surprisingly little attention has been given to examining the datasets that these benchmarks rely on. This survey addresses that gap by presenting a broad and careful review of the most widely used fairness datasets in current language model research, characterizing them along several key dimensions including their origin, scope, content, and intended use to help researchers better appreciate the assumptions and limitations embedded in these resources. To support more meaningful comparisons and analyses, we introduce a unified evaluation framework that reveals consistent patterns of demographic disparities across datasets and scoring methods. Applying this framework to twenty four common benchmarks, we highlight the often overlooked biases that can influence conclusions about model fairness and offer practical guidance for selecting, combining, and interpreting these datasets. We also point to opportunities for creating new fairness benchmarks that reflect more diverse social contexts and encourage more thoughtful use of these tools going forward. All code, data, and detailed results are publicly available at https://github.com/vanbanTruong/Fairness-in-Large-Language-Models/tree/main/datasets to promote transparency and reproducibility across the research community.

Problem

• Examining fairness datasets used in language model benchmarks
• Identifying biases in datasets affecting fairness conclusions
• Proposing a unified framework for evaluating demographic disparities

Innovation

• Surveying widely used fairness datasets in language models
• Introducing unified framework for demographic disparity analysis
• Providing guidance for dataset selection and bias mitigation