FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filtering

2026-07-05

AI Summary

This work addresses the challenges of model drift and degraded generalization in federated learning caused by heterogeneous client data, exacerbated by Sharpness-Aware Minimization (SAM) due to inconsistent local perturbations across clients that lead to optimization divergence. For the first time, the authors analyze SAM perturbations from a frequency-domain perspective and reveal that their inconsistency predominantly resides in low-frequency components. Building on this insight, they propose FedFFT, a lightweight, plug-and-play method that applies low-pass filtering to SAM perturbations without increasing communication overhead, effectively suppressing harmful updates while preserving shared learning signals. Extensive experiments demonstrate that FedFFT consistently outperforms existing federated SAM approaches across various non-IID settings, achieving superior robustness, generalization, and convergence stability.

Abstract

Federated Learning (FL) enables decentralized training without data sharing, but suffers from statistical heterogeneity across clients, leading to client drift, poor generalization, and sharp minima compared to centralized training. Sharpness-Aware Minimization (SAM) has emerged as a promising approach to improve generalization, yet its application in federated learning still suffers from divergence problems, since perturbations are computed locally and reflect client-specific loss geometries. To better understand this issue, we provide experimental evidence from a new perspective, the frequency domain, for SAM perturbations in federated settings, revealing that inter-client perturbation inconsistencies are predominantly concentrated in the low-frequency spectrum. Motivated by this insight, we propose Federated learning with Frequency-domain Filtering of SAM perturbations (FedFFT). It is a lightweight and plug-and-play method that filters out low-frequency components of SAM perturbations without requiring additional communication, thereby suppressing inconsistent components in client updates while preserving consistent learning signals. Extensive experiments across multiple benchmarks and diverse backbones demonstrate that FedFFT consistently outperforms SAM-based FL methods, particularly under severe non-IID distributions. These results highlight the effectiveness, scalability, and general applicability of our frequency-domain perspective for sharpness-aware federated optimization.

Problem

• Federated Learning
• Client Drift
• Sharpness-Aware Minimization
• Statistical Heterogeneity
• Perturbation Inconsistency

Innovation

• Federated Learning
• Sharpness-Aware Minimization
• Frequency-domain Filtering
• Client Drift
• Spectral Perturbation