Discovering Collaboration from Novelty: Random Network Distillation for Clustered Federated Learning

2026-06-29

AI Summary

This work addresses the performance limitations of federated learning under non-IID data, where a single global model struggles to accommodate client heterogeneity. Existing clustering approaches often intertwine clustering with model training, incurring substantial communication and computational overhead. To overcome this, we propose a lightweight, decoupled client clustering method that introduces Random Network Distillation (RND) into federated learning for the first time. By leveraging locally computed RND prediction errors as task-agnostic novelty signals, our approach constructs a similarity metric that enables pre-training clustering without sharing raw data or repeatedly evaluating the main model. This facilitates runtime discovery of semantically coherent collaboration groups, significantly reducing communication and computation costs while enhancing model generalization and deployment efficiency in non-IID settings.

Abstract

Federated Learning often suffers under non-independently and identically distributed data, where a single global model may fail to represent the diversity of client distributions. Clustered Federated Learning mitigates this issue by training specialized models for groups of similar clients, but existing approaches often couple cluster assignment with the main training loop, increasing computational and communication costs. We propose a lightweight clustering approach based on Random Network Distillation. Each client trains a compact Random Network Distillation predictor on its local data and uses its prediction error as a novelty signal to estimate similarity with other clients. This enables the discovery of meaningful client groups before federated training, without sharing raw data or repeatedly evaluating the main model. Crucially, the resulting federations emerge from local novelty estimates at runtime, making the method suitable for autonomous large-scale distributed systems where neither the number of clusters nor the collaboration structure can be specified a priori. Overall, by decoupling clustering from learning, the method provides a task-agnostic and efficient mechanism for autonomous collaboration under non-independently and identically distributed data.

Problem

• Federated Learning
• Non-IID data
• Clustered Federated Learning
• Client Collaboration
• Autonomous Clustering

Innovation

• Random Network Distillation
• Clustered Federated Learning
• Non-IID Data
• Autonomous Collaboration
• Novelty-based Clustering