How fair are we? From conceptualization to automated assessment of fairness definitions

2024-04-15

arXiv.org

AI Summary

Existing automated fairness-checking approaches for software rely on predefined rules and lack support for user-defined fairness criteria—particularly challenging in emerging domains such as software engineering (SE) recommendation systems and Arduino hardware component recommendation, where bias mitigation demands domain-specific adaptability. Method: We propose MODNESS, a model-driven framework built upon Model-Driven Engineering (MDE) and a Domain-Specific Modeling Language (DSML), enabling users to flexibly specify fairness concepts, compose multi-dimensional fairness metrics, and automatically generate executable assessment code within a dedicated modeling environment. Contribution/Results: MODNESS is the first framework to enable user-driven fairness modeling and code generation; it pioneers fairness evaluation in SE and Arduino component recommendation—two previously unaddressed application domains; and it overcomes expressiveness and domain-adaptation limitations of prior tools. Empirical evaluation demonstrates end-to-end fairness assessment capability, significantly improving flexibility and cross-domain applicability.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Fairness is a critical concept in ethics and social domains, but it is also a challenging property to engineer in software systems. With the increasing use of machine learning in software systems, researchers have been developing techniques to automatically assess the fairness of software systems. Nonetheless, a significant proportion of these techniques rely upon pre-established fairness definitions, metrics, and criteria, which may fail to encompass the wide-ranging needs and preferences of users and stakeholders. To overcome this limitation, we propose a novel approach, called MODNESS, that enables users to customize and define their fairness concepts using a dedicated modeling environment. Our approach guides the user through the definition of new fairness concepts also in emerging domains, and the specification and composition of metrics for its evaluation. Ultimately, MODNESS generates the source code to implement fair assessment based on these custom definitions. In addition, we elucidate the process we followed to collect and analyze relevant literature on fairness assessment in software engineering (SE). We compare MODNESS with the selected approaches and evaluate how they support the distinguishing features identified by our study. Our findings reveal that i) most of the current approaches do not support user-defined fairness concepts; ii) our approach can cover two additional application domains not addressed by currently available tools, i.e., mitigating bias in recommender systems for software engineering and Arduino software component recommendations; iii) MODNESS demonstrates the capability to overcome the limitations of the only two other Model-Driven Engineering-based approaches for fairness assessment.

Problem

• Software Fairness
• Customizable Standards
• Bias in Recommendations

Innovation

• Customizable Fairness Standards
• Bias Detection in Software
• Model-Driven Engineering for Fairness