An Efficient Gradient-Aware Error-Bounded Lossy Compressor for Federated Learning

2025-11-07

AI Summary

In federated learning (FL), high communication overhead from gradient transmission severely limits scalability in bandwidth-constrained settings. Existing error-bounded lossy compression methods (e.g., SZ3), designed for smooth scientific data, suffer from low compression ratios and substantial accuracy degradation when applied to FL gradients—characterized by low spatial correlation. To address this, we propose the first error-bounded lossy compression (EBLC) framework specifically tailored to FL gradient properties. Our method introduces two novel predictors: (i) a cross-round exponential moving average (EMA) magnitude predictor, and (ii) a sign predictor leveraging gradient oscillation patterns and kernel-level sign consistency—both significantly reducing residual entropy. Integrated with normalized EMA, sign modeling, and standard entropy coding, the framework achieves end-to-end EBLC. Experiments show a 1.53× higher compression ratio than SZ3 with lower accuracy loss, reducing end-to-end communication time by 76.1%–96.2% in APPFL.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Learning & Optimization for NLP
• Data Mining & Knowledge Management: Data Compression

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated learning (FL) enables collaborative model training without exposing clients'private data, but its deployment is often constrained by the communication cost of transmitting gradients between clients and the central server, especially under system heterogeneity where low-bandwidth clients bottleneck overall performance. Lossy compression of gradient data can mitigate this overhead, and error-bounded lossy compression (EBLC) is particularly appealing for its fine-grained utility-compression tradeoff. However, existing EBLC methods (e.g., SZ), originally designed for smooth scientific data with strong spatial locality, rely on generic predictors such as Lorenzo and interpolation for entropy reduction to improve compression ratio. Gradient tensors, in contrast, exhibit low smoothness and weak spatial correlation, rendering these predictors ineffective and leading to poor compression ratios. To address this limitation, we propose an EBLC framework tailored for FL gradient data to achieve high compression ratios while preserving model accuracy. The core of it is an innovative prediction mechanism that exploits temporal correlations across FL training rounds and structural regularities within convolutional kernels to reduce residual entropy. The predictor is compatible with standard quantizers and entropy coders and comprises (1) a cross-round magnitude predictor based on a normalized exponential moving average, and (2) a sign predictor that leverages gradient oscillation and kernel-level sign consistency. Experiments show that this new EBLC yields up to 1.53x higher compression ratios than SZ3 with lower accuracy loss. Integrated into a real-world FL framework, APPFL, it reduces end-to-end communication time by 76.1%-96.2% under various constrained-bandwidth scenarios, demonstrating strong scalability for real-world FL deployments.

Problem

• Compressing gradients in federated learning to reduce communication costs
• Addressing poor compression ratios from existing methods on non-smooth gradient data
• Maintaining model accuracy while achieving high compression through temporal correlations

Innovation

• Exploits temporal correlations across training rounds
• Leverages structural regularities in convolutional kernels
• Uses cross-round magnitude and sign predictors