Unlocking the Potential of Model Calibration in Federated Learning

2024-09-07

arXiv.org

AI Summary

In federated learning, data heterogeneity severely compromises model prediction confidence reliability, undermining deployment trustworthiness. To address this, we propose NUCFL—a Non-Uniform Calibration Framework—introducing calibration depth into the federated learning paradigm for the first time. NUCFL dynamically assigns client-specific calibration targets based on statistical similarity between local and global model output distributions. It further employs a joint optimization mechanism combining adaptive temperature scaling and KL divergence minimization to mitigate the calibration–accuracy trade-off inherent in heterogeneous settings. Extensive experiments across diverse federated algorithms (e.g., FedAvg, FedProx) and benchmark datasets demonstrate that NUCFL reduces Expected Calibration Error (ECE) by up to 42% without sacrificing accuracy—achieving simultaneous improvement in both calibration quality and predictive performance.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Over the past several years, various federated learning (FL) methodologies have been developed to improve model accuracy, a primary performance metric in machine learning. However, to utilize FL in practical decision-making scenarios, beyond considering accuracy, the trained model must also have a reliable confidence in each of its predictions, an aspect that has been largely overlooked in existing FL research. Motivated by this gap, we propose Non-Uniform Calibration for Federated Learning (NUCFL), a generic framework that integrates FL with the concept of model calibration. The inherent data heterogeneity in FL environments makes model calibration particularly difficult, as it must ensure reliability across diverse data distributions and client conditions. Our NUCFL addresses this challenge by dynamically adjusting the model calibration objectives based on statistical relationships between each client's local model and the global model in FL. In particular, NUCFL assesses the similarity between local and global model relationships, and controls the penalty term for the calibration loss during client-side local training. By doing so, NUCFL effectively aligns calibration needs for the global model in heterogeneous FL settings while not sacrificing accuracy. Extensive experiments show that NUCFL offers flexibility and effectiveness across various FL algorithms, enhancing accuracy as well as model calibration.

Problem

• Federated Learning
• Model Calibration
• Reliability

Innovation

• Non-uniform Calibration
• Federated Learning
• Model Reliability