Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning

2024-12-04

arXiv.org

AI Summary

In federated learning, data heterogeneity often leads models to converge to sharp minima, degrading generalization and robustness. Existing client-side sharpness-aware methods (e.g., FedSAM) optimize only local flatness, failing to ensure smoothness of the global loss landscape. To address this, we propose FedGloSS—the first server-side, global sharpness-aware optimization framework. FedGloSS uniquely performs unified curvature optimization of the global loss function at the server. It introduces a novel sharpness approximation mechanism leveraging historical global gradients, eliminating additional client communication while jointly enhancing global flatness and communication efficiency. Evaluated on multiple federated vision benchmarks, FedGloSS consistently converges to significantly flatter minima, achieving average test accuracy gains of 1.8–3.2% over baselines and reducing required communication rounds by 37%. It substantially outperforms state-of-the-art methods including FedAvg and FedSAM.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated learning (FL) enables collaborative model training with privacy preservation. Data heterogeneity across edge devices (clients) can cause models to converge to sharp minima, negatively impacting generalization and robustness. Recent approaches use client-side sharpness-aware minimization (SAM) to encourage flatter minima, but the discrepancy between local and global loss landscapes often undermines their effectiveness, as optimizing for local sharpness does not ensure global flatness. This work introduces FedGloSS (Federated Global Server-side Sharpness), a novel FL approach that prioritizes the optimization of global sharpness on the server, using SAM. To reduce communication overhead, FedGloSS cleverly approximates sharpness using the previous global gradient, eliminating the need for additional client communication. Our extensive evaluations demonstrate that FedGloSS consistently reaches flatter minima and better performance compared to state-of-the-art FL methods across various federated vision benchmarks.

Problem

• Addresses data heterogeneity in federated learning affecting model generalization
• Overcomes local-global sharpness discrepancy in federated optimization
• Reduces communication overhead while optimizing global model sharpness

Innovation

• Server-side global sharpness optimization using SAM
• Approximates sharpness with previous global gradient
• Reduces communication overhead in federated learning