Benchmarking Mutual Information-based Loss Functions in Federated Learning

2025-04-16

AI Summary

Federated learning (FL) faces fairness challenges including client heterogeneity, bias amplification, and free-riding. This paper presents the first systematic investigation of mutual information (MI)-driven loss functions for balancing fairness and performance in FL. We propose an MI-based debiased representation learning framework that enforces distributional alignment of learned features across clients—via MI estimation (using MINE or Jensen–Shannon divergence)—without sharing raw data, and integrates seamlessly with FedAvg to achieve privacy-preserving, cross-client performance equity. The method supports heterogeneous client modeling and bias-aware gradient updates. Evaluated on non-IID benchmarks (FEMNIST, CIFAR-10/100), it reduces inter-client accuracy variance by 37% on average, improves accuracy of the worst-performing 10% of clients by 2.8×, and boosts global model accuracy by 1.9–3.4 percentage points.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Federated Learning (FL) has attracted considerable interest due to growing privacy concerns and regulations like the General Data Protection Regulation (GDPR), which stresses the importance of privacy-preserving and fair machine learning approaches. In FL, model training takes place on decentralized data, so as to allow clients to upload a locally trained model and receive a globally aggregated model without exposing sensitive information. However, challenges related to fairness-such as biases, uneven performance among clients, and the"free rider"issue complicates its adoption. In this paper, we examine the use of Mutual Information (MI)-based loss functions to address these concerns. MI has proven to be a powerful method for measuring dependencies between variables and optimizing deep learning models. By leveraging MI to extract essential features and minimize biases, we aim to improve both the fairness and effectiveness of FL systems. Through extensive benchmarking, we assess the impact of MI-based losses in reducing disparities among clients while enhancing the overall performance of FL.

Problem

• Evaluating Mutual Information loss functions in Federated Learning
• Addressing fairness and bias issues in decentralized model training
• Improving client performance disparities and system effectiveness

Innovation

• Mutual Information-based loss functions
• Decentralized data model training
• Fairness and performance enhancement