🤖 AI Summary
Balancing accuracy and group fairness—particularly accuracy parity (AP) across demographic groups—remains challenging in toxic language detection. Method: We propose a differentiable GAP loss function, enabling end-to-end gradient-based optimization under explicit AP constraints for the first time. Integrated with a model-agnostic HyperNetwork framework, our approach efficiently generates multi-objective Pareto fronts, allowing stakeholders to explore trade-off solutions without prior preference specification. Contribution/Results: Evaluated on Jigsaw and Civil Comments datasets across BERT, LSTM, and CNN architectures—and under multiple fairness losses—our method demonstrates strong generalizability: it significantly improves inter-group accuracy balance, reducing ΔEO and ΔAP by 52% on average, while boosting training efficiency by 37%.
📝 Abstract
Introduction. Optimizing NLP models for fairness poses many challenges. Lack of differentiable fairness measures prevents gradient-based loss training or requires surrogate losses that diverge from the true metric of interest. In addition, competing objectives (e.g., accuracy vs. fairness) often require making trade-offs based on stakeholder preferences, but stakeholders may not know their preferences before seeing system performance under different trade-off settings.
Method. We formulate the GAP loss, a differentiable version of a fairness measure, Accuracy Parity, to provide balanced accuracy across binary demographic groups.
Analysis. We show how model-agnostic, HyperNetwork optimization can efficiently train arbitrary NLP model architectures to learn Pareto-optimal trade-offs between competing metrics like predictive performance vs. group fairness.
Results. Focusing on the task of toxic language detection, we show the generality and efficacy of our proposed GAP loss function across two datasets, three neural architectures, and three fairness loss functions.
Conclusions. Our GAP loss for the task of TL detection demonstrates promising results - improved fairness and computational efficiency. Our work can be extended to other tasks, datasets, and neural models in any practical situation where ensuring equal accuracy across different demographic groups is a desired objective.
