TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments

2025-07-06

AI Summary

In resource-constrained edge federated learning, prototype-based federated learning (PFL) suffers from prohibitively high communication overhead, which scales quadratically with feature dimensionality and number of classes. To address this, we propose Class Prototype Sparsification with Adaptive Scaling (CPS-AS), a communication-efficient PFL framework that imposes structured sparsity at the class-prototype level, selectively transmits only non-zero prototype elements, and adaptively scales prototypes based on local class distributions—all without imposing additional computational burden on clients. CPS-AS thus achieves efficient, heterogeneity-aware prototype compression. Extensive experiments across multiple benchmark datasets demonstrate up to 4× reduction in communication volume, with negligible accuracy degradation (<0.5% drop), significantly outperforming existing PFL communication compression methods. The approach is particularly suitable for bandwidth-limited, device-heterogeneous edge federated deployments.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Learning & Optimization for CV
• Multiagent Systems: Agent Communication

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Communication efficiency in federated learning (FL) remains a critical challenge for resource-constrained environments. While prototype-based FL reduces communication overhead by sharing class prototypes-mean activations in the penultimate layer-instead of model parameters, its efficiency decreases with larger feature dimensions and class counts. We propose TinyProto, which addresses these limitations through Class-wise Prototype Sparsification (CPS) and adaptive prototype scaling. CPS enables structured sparsity by allocating specific dimensions to class prototypes and transmitting only non-zero elements, while adaptive scaling adjusts prototypes based on class distributions. Our experiments show TinyProto reduces communication costs by up to 4x compared to existing methods while maintaining performance. Beyond its communication efficiency, TinyProto offers crucial advantages: achieving compression without client-side computational overhead and supporting heterogeneous architectures, making it ideal for resource-constrained heterogeneous FL.

Problem

• Reduces communication costs in federated learning
• Addresses inefficiency in prototype-based FL with large features
• Supports resource-constrained and heterogeneous FL environments

Innovation

• Class-wise Prototype Sparsification reduces dimensions
• Adaptive scaling adjusts prototypes dynamically
• Sparse prototypes cut communication costs significantly