fairmetrics: An R package for group fairness evaluation

2025-06-06

AI Summary

This study addresses systemic bias in machine learning models against protected groups (e.g., race, gender, age). We propose and implement the first open-source R package dedicated to group fairness assessment. The package comprehensively supports three foundational fairness criteria—independence, separation, and sufficiency—and unifies, for the first time in the R ecosystem, point estimation and bootstrap confidence interval computation for over 15 mainstream fairness metrics (e.g., statistical parity, equal opportunity, predictive equality). It provides a standardized API and includes illustrative examples using real-world clinical data (MIMIC-II). Our key contribution lies in rigorously integrating statistical inference with fairness measurement theory, substantially lowering the barrier to quantitative fairness analysis. The package is publicly available on CRAN and GitHub, and has already gained early adoption within the R community and academic citations.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Fairness is a growing area of machine learning (ML) that focuses on ensuring models do not produce systematically biased outcomes for specific groups, particularly those defined by protected attributes such as race, gender, or age. Evaluating fairness is a critical aspect of ML model development, as biased models can perpetuate structural inequalities. The {fairmetrics} R package offers a user-friendly framework for rigorously evaluating numerous group-based fairness criteria, including metrics based on independence (e.g., statistical parity), separation (e.g., equalized odds), and sufficiency (e.g., predictive parity). Group-based fairness criteria assess whether a model is equally accurate or well-calibrated across a set of predefined groups so that appropriate bias mitigation strategies can be implemented. {fairmetrics} provides both point and interval estimates for multiple metrics through a convenient wrapper function and includes an example dataset derived from the Medical Information Mart for Intensive Care, version II (MIMIC-II) database (Goldberger et al., 2000; Raffa, 2016).

Problem

• Evaluating group fairness in ML models to prevent biased outcomes
• Providing metrics for fairness criteria like independence and separation
• Offering tools for bias mitigation across protected attribute groups

Innovation

• R package for group fairness evaluation
• Evaluates independence, separation, sufficiency metrics
• Provides point and interval estimates conveniently