E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing

2025-02-05

AI Summary

To address the prohibitive communication overhead induced by large models in federated learning, this paper proposes the One-Step Synthetic Feature Compression (3SFC) framework. 3SFC introduces a novel gradient compression paradigm wherein the client model itself serves as the decompressor, integrating inverse gradient reconstruction guided by weight and objective-function priors, one-step synthetic feature generation, error feedback, and dual-path expansion (E-3SFC), enabling dynamic communication budget scheduling. We provide theoretical convergence guarantees for both convex and non-convex settings. Extensive experiments across six benchmark datasets and six model architectures demonstrate that 3SFC outperforms state-of-the-art methods by up to 13.4% in accuracy while reducing total communication volume by up to 111.6×. The framework thus achieves substantial improvements in communication efficiency without compromising model performance.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The exponential growth in model sizes has significantly increased the communication burden in Federated Learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed Extended Single-Step Synthetic Features Compressing (E-3SFC), which consists of three sub-components, i.e., the Single-Step Synthetic Features Compressor (3SFC), a double-way compression algorithm, and a communication budget scheduler. First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing double-way compression and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and non-convex conditions demonstrates that 3SFC achieves linear and sub-linear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

Problem

• Addresses communication burden in Federated Learning
• Proposes E-3SFC for efficient gradient compression
• Reduces communication costs without performance loss

Innovation

• Double-way gradient compression
• Single-step synthetic features
• Dynamic communication scheduling