Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs

2025-05-29

AI Summary

Conventional fairness metrics rely solely on prediction accuracy, overlooking implicit decision biases arising from model uncertainty—particularly inter-group disparities in confidence scores. Method: We propose UCerF, the first uncertainty-aware fairness evaluation metric for large language models (LLMs), integrating group-wise confidence statistics and probabilistic calibration into fairness quantification to expose latent biases—such as high-confidence erroneous predictions—that metrics like Equalized Odds neglect. To support this, we construct the first fine-grained, LLM-optimized gender-occupation coreference resolution benchmark (31,756 instances) with controllable bias injection. Results: Empirical evaluation across 10 mainstream open-source LLMs demonstrates that UCerF significantly enhances sensitivity to and interpretability of intrinsic decision biases—for instance, revealing covert unfairness in Mistral-7B stemming from overconfident incorrect predictions.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Bias, Fairness & Privacy
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

The recent rapid adoption of large language models (LLMs) highlights the critical need for benchmarking their fairness. Conventional fairness metrics, which focus on discrete accuracy-based evaluations (i.e., prediction correctness), fail to capture the implicit impact of model uncertainty (e.g., higher model confidence about one group over another despite similar accuracy). To address this limitation, we propose an uncertainty-aware fairness metric, UCerF, to enable a fine-grained evaluation of model fairness that is more reflective of the internal bias in model decisions compared to conventional fairness measures. Furthermore, observing data size, diversity, and clarity issues in current datasets, we introduce a new gender-occupation fairness evaluation dataset with 31,756 samples for co-reference resolution, offering a more diverse and suitable dataset for evaluating modern LLMs. We establish a benchmark, using our metric and dataset, and apply it to evaluate the behavior of ten open-source LLMs. For example, Mistral-7B exhibits suboptimal fairness due to high confidence in incorrect predictions, a detail overlooked by Equalized Odds but captured by UCerF. Overall, our proposed LLM benchmark, which evaluates fairness with uncertainty awareness, paves the way for developing more transparent and accountable AI systems.

Problem

• Assessing fairness in LLMs with uncertainty-aware metrics
• Addressing dataset limitations for gender-occupation fairness evaluation
• Benchmarking LLM fairness using UCerF and a new diverse dataset

Innovation

• Proposes UCerF for uncertainty-aware fairness evaluation
• Introduces diverse gender-occupation fairness dataset
• Benchmarks ten LLMs using new metric and dataset