ColNet: Collaborative Optimization in Decentralized Federated Multi-task Learning Systems

2025-01-17

AI Summary

Decentralized federated learning faces significant challenges due to inherent task heterogeneity across clients and the absence of a central coordinating server. Method: This paper proposes ColNet, a collaborative optimization framework that jointly addresses these issues through four key innovations: (i) model partitioning—sharing a common backbone while retaining client-specific task heads; (ii) client similarity-based clustering; (iii) a decentralized group leadership mechanism; and (iv) conflict-aware cross-group parameter aggregation—all operating in a serverless architecture. Contribution/Results: ColNet is the first approach to systematically tackle task heterogeneity in fully decentralized settings. It enables simultaneous backbone sharing and task-head personalization, achieving faster convergence and superior generalization. Extensive experiments under diverse label- and task-heterogeneous federated configurations demonstrate consistent and significant improvements over state-of-the-art decentralized aggregation methods.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The integration of Federated Learning (FL) and Multi-Task Learning (MTL) has been explored to address client heterogeneity, with Federated Multi-Task Learning (FMTL) treating each client as a distinct task. However, most existing research focuses on data heterogeneity (e.g., addressing non-IID data) rather than task heterogeneity, where clients solve fundamentally different tasks. Additionally, much of the work relies on centralized settings with a server managing the federation, leaving the more challenging domain of decentralized FMTL largely unexplored. Thus, this work bridges this gap by proposing ColNet, a framework designed for heterogeneous tasks in decentralized federated environments. ColNet divides models into the backbone and task-specific layers, forming groups of similar clients, with group leaders performing conflict-averse cross-group aggregation. A pool of experiments with different federations demonstrated ColNet outperforms the compared aggregation schemes in decentralized settings with label and task heterogeneity scenarios.

Problem

• Federated Learning
• Multi-task Learning
• Decentralized Control

Innovation

• ColNet
• Decentralized Multi-task Learning
• Group Leader Coordination Mechanism