Revisiting Pre-processing Group Fairness: A Modular Benchmarking Framework

2025-08-20

AI Summary

Fairness pre-processing methods offer model-agnosticism and privacy preservation but suffer from poor comparability and reproducibility due to the absence of standardized evaluation frameworks. To address this, we propose FairPrep—the first modular, extensible benchmarking framework for fairness pre-processing on tabular data. Built upon AIF360, FairPrep unifies datasets, pre-processing algorithms, and downstream models into a cohesive pipeline. It enables automated large-scale experiments and jointly evaluates multidimensional fairness metrics (e.g., statistical parity, equal opportunity) alongside utility measures (e.g., accuracy, F1-score), generating standardized, interpretable reports. By providing a unified, open-source infrastructure, FairPrep bridges the critical gap in empirical evaluation of data-level fairness interventions. It significantly enhances experimental reproducibility and cross-method comparability, establishing a practical, community-oriented benchmark for fair machine learning research.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

As machine learning systems become increasingly integrated into high-stakes decision-making processes, ensuring fairness in algorithmic outcomes has become a critical concern. Methods to mitigate bias typically fall into three categories: pre-processing, in-processing, and post-processing. While significant attention has been devoted to the latter two, pre-processing methods, which operate at the data level and offer advantages such as model-agnosticism and improved privacy compliance, have received comparatively less focus and lack standardised evaluation tools. In this work, we introduce FairPrep, an extensible and modular benchmarking framework designed to evaluate fairness-aware pre-processing techniques on tabular datasets. Built on the AIF360 platform, FairPrep allows seamless integration of datasets, fairness interventions, and predictive models. It features a batch-processing interface that enables efficient experimentation and automatic reporting of fairness and utility metrics. By offering standardised pipelines and supporting reproducible evaluations, FairPrep fills a critical gap in the fairness benchmarking landscape and provides a practical foundation for advancing data-level fairness research.

Problem

• Evaluating fairness-aware pre-processing techniques for tabular datasets
• Addressing lack of standardized benchmarking tools for data-level fairness methods
• Providing reproducible evaluation framework for model-agnostic bias mitigation approaches

Innovation

• Modular framework for fairness pre-processing
• Batch-processing interface for efficient experimentation
• Standardized pipelines for reproducible fairness evaluations