ProFL: Performative Robust Optimal Federated Learning

2024-10-23

arXiv.org

AI Summary

This work addresses the deployment-induced data distribution shift—termed *executive prediction*—in federated learning, under realistic conditions involving noisy, corrupted, and heterogeneous client data. We propose the first distributed algorithm provably convergent to an *executive-optimal solution* (not merely a stationary point). Methodologically, we relax the strong assumptions of convex risk functions and noise-free data adopted in prior work, instead integrating robust gradient estimation, distributed momentum correction, and non-convex convergence analysis grounded in the Polyak–Łojasiewicz (PL) condition, while supporting multi-client asynchronous updates. Experiments on multiple benchmark datasets demonstrate that our algorithm significantly outperforms state-of-the-art methods: it converges to superior executive-optimal solutions and exhibits strong robustness and practicality under both data noise and distribution drift.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Performative prediction is a framework that captures distribution shifts that occur during the training of machine learning models due to their deployment. As the trained model is used, data generation causes the model to evolve, leading to deviations from the original data distribution. The impact of such model-induced distribution shifts in federated learning is increasingly likely to transpire in real-life use cases. A recently proposed approach extends performative prediction to federated learning with the resulting model converging to a performative stable point, which may be far from the performative optimal point. Earlier research in centralized settings has shown that the performative optimal point can be achieved under model-induced distribution shifts, but these approaches require the performative risk to be convex and the training data to be noiseless, assumptions often violated in realistic federated learning systems. This paper overcomes all of these shortcomings and proposes Performative Robust Optimal Federated Learning, an algorithm that finds performative optimal points in federated learning from noisy and contaminated data. We present the convergence analysis under the Polyak-Lojasiewicz condition, which applies to non-convex objectives. Extensive experiments on multiple datasets demonstrate the advantage of Robust Optimal Federated Learning over the state-of-the-art.

Problem

• Addresses model-induced distribution shifts in federated learning
• Overcomes limitations of convexity and noiseless data assumptions
• Finds performative optimal points from noisy and contaminated data

Innovation

• Federated learning with model-induced distribution shifts
• Achieving performative optimal points from noisy data
• Convergence under non-convex Polyak-Lojasiewicz condition