TurboSVM-FL: Boosting Federated Learning through SVM Aggregation for Lazy Clients

2024-01-22

AAAI Conference on Artificial Intelligence

AI Summary

To address slow convergence in cross-device federated learning caused by data heterogeneity—particularly under realistic constraints of limited client computation/storage and few local iterations (“lazy” clients)—this paper proposes a novel aggregation strategy imposing zero client-side overhead. The core innovation is the first integration of support vector machines (SVMs) into federated aggregation: leveraging selective class embedding aggregation and max-margin spread-out regularization at the server, it implicitly models class-discriminative decision boundaries without modifying clients’ local training procedures. Evaluated on non-IID benchmarks—including FEMNIST, CelebA, and Shakespeare—the method significantly accelerates convergence, reduces communication rounds, and simultaneously improves accuracy, F1 score, and Matthews Correlation Coefficient (MCC).

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Federated learning is a distributed collaborative machine learning paradigm that has gained strong momentum in recent years. In federated learning, a central server periodically coordinates models with clients and aggregates the models trained locally by clients without necessitating access to local data. Despite its potential, the implementation of federated learning continues to encounter several challenges, predominantly the slow convergence that is largely due to data heterogeneity. The slow convergence becomes particularly problematic in cross-device federated learning scenarios where clients may be strongly limited by computing power and storage space, and hence counteracting methods that induce additional computation or memory cost on the client side such as auxiliary objective terms and larger training iterations can be impractical. In this paper, we propose a novel federated aggregation strategy, TurboSVM-FL, that poses no additional computation burden on the client side and can significantly accelerate convergence for federated classification task, especially when clients are "lazy" and train their models solely for few epochs for next global aggregation. TurboSVM-FL extensively utilizes support vector machine to conduct selective aggregation and max-margin spread-out regularization on class embeddings. We evaluate TurboSVM-FL on multiple datasets including FEMNIST, CelebA, and Shakespeare using user-independent validation with non-iid data distribution. Our results show that TurboSVM-FL can significantly outperform existing popular algorithms on convergence rate and reduce communication rounds while delivering better test metrics including accuracy, F1 score, and MCC.

Problem

• Addresses slow convergence in federated learning due to data heterogeneity
• Reduces computation burden on clients with limited resources
• Improves classification accuracy and convergence rate using SVM aggregation

Innovation

• SVM-based selective aggregation for lazy clients
• Max-margin spread-out regularization on embeddings
• No extra client-side computation burden