Lossy Compression for Sparse Aggregation

2026-06-29

AI Summary

This work addresses the trade-off between communication overhead and aggregation accuracy in distributed learning when transmitting sparse local models. It proposes the first compression scheme that integrates covering codes with sketching techniques and establishes a dual information-theoretic lower bound based on f-divergence. This bound is tight for binary alphabets and strictly stronger than conventional Fano-type bounds. The proposed method achieves a communication–accuracy trade-off approaching the theoretical limit in frequency estimation tasks. Although it does not attain the bound for general alphabets, it opens a new avenue for future research in this direction.

Abstract

We consider the problem of transmitting sparse local updates to the server in a distributed learning system. Specifically, the system consists of $n$ clients, each possessing a $k$-sparse $d$-dimensional local model, and a central server responsible for aggregating the clients' models into a global model. The goal is to characterize the tradeoff between the communication cost in the transmission from the clients to the server and the accuracy in aggregating the global model. We propose a compression scheme for sparse local models by concatenating a covering method and a sketching method. We also present a converse based on f-divergence, which strengthens the conventional Fano-type lower bounds. The proposed lower bound is tight for the frequency estimation case, that is, each coordinate takes values in a binary alphabet. For general alphabets, the proposed achievable schemes remain suboptimal relative to the converse bounds, indicating that a complete characterization of the communication-accuracy tradeoff requires further investigation.

Problem

• lossy compression
• sparse aggregation
• distributed learning
• communication-accuracy tradeoff
• f-divergence

Innovation

• lossy compression
• sparse aggregation
• covering method
• sketching method
• f-divergence