A Hierarchical Sampling Framework for bounding the Generalization Error of Federated Learning

2026-05-05

AI Summary

This work addresses the problem of characterizing tight generalization error bounds in hierarchical federated learning when data exhibits hierarchical dependency structures. The authors propose a Wasserstein distance–based hierarchical sampling framework that models inter-client data dependencies through a tree structure and analyzes algorithmic sensitivity to single-node perturbations via an auxiliary sample construction. This approach yields a significantly tighter upper bound on the generalization error. Notably, this is the first study to incorporate Wasserstein distance into hierarchical sampling settings, rigorously extending existing conditional mutual information (CMI)–based bounds. Furthermore, the framework naturally integrates with differential privacy to derive privacy-aware generalization bounds. Under the Gaussian location model, the derived bound accurately recovers the asymptotic rate of the true generalization error, demonstrating its theoretical sharpness and practical relevance.

Abstract

We study expected generalization bounds for the Hierarchical Federated Learning (HFL) setup using Wasserstein distance. We introduce a generalized framework in which data is sampled hierarchically, and we model it with a multi-layered tree structure that induces dependencies among the clients' datasets. We derive generalization bounds in terms of Wasserstein distance under the Lipschitz assumption on the loss function, by applying a supersample construction that allows us to measure the sensitivity of the algorithm to the change of a single node in the sampling tree. By leveraging the FL structure, we recover and strictly imply existing state-of-the-art conditional mutual information (CMI) bounds in the case of bounded losses. We also show that our bound can be applied together with Differential Privacy assumptions, to recover generalization bounds based on algorithmic privacy. To assess the tightness of our bounds, we study the Gaussian Location Model (GLM) and show that we recover the actual asymptotic rate of the generalization error.

Problem

• Generalization Error
• Federated Learning
• Hierarchical Sampling
• Wasserstein Distance
• Data Dependency

Innovation

• Hierarchical Federated Learning
• Wasserstein distance
• Generalization bound
• Supersample construction
• Conditional Mutual Information