Traversal Learning Coordination For Lossless And Efficient Distributed Learning

2025-04-10

AI Summary

To address model performance degradation in federated learning (FL), split learning (SL), and split federated learning (SFL) caused by averaging-based aggregation or gradient splitting, this paper proposes Traversal Learning (TL): a lossless training paradigm featuring forward distributed traversal and backward centralized optimization. Its core innovation lies in the forward phase—where the model sequentially traverses each node to perform local forward propagation—while backward propagation is exclusively executed by a central coordinator. Gradient consistency is rigorously preserved via virtual batch construction and data index alignment, fundamentally mitigating aggregation distortion and split-gradient bias. Evaluated across six cross-domain datasets—including IID/non-IID image, text, healthcare, and financial data—TL consistently outperforms state-of-the-art methods (e.g., FedAvg, SplitNN) in accuracy, macro-F1, and AUC, achieving up to 7.85% improvement in IID accuracy and 4.54% in financial AUC, closely approaching centralized training performance.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In this paper, we introduce Traversal Learning (TL), a novel approach designed to address the problem of decreased quality encountered in popular distributed learning (DL) paradigms such as Federated Learning (FL), Split Learning (SL), and SplitFed Learning (SFL). Traditional FL experiences from an accuracy drop during aggregation due to its averaging function, while SL and SFL face increased loss due to the independent gradient updates on each split network. TL adopts a unique strategy where the model traverses the nodes during forward propagation (FP) and performs backward propagation (BP) on the orchestrator, effectively implementing centralized learning (CL) principles within a distributed environment. The orchestrator is tasked with generating virtual batches and planning the sequential node visits of the model during FP, aligning them with the ordered index of the data within these batches. We conducted experiments on six datasets representing diverse characteristics across various domains. Our evaluation demonstrates that TL is on par with classic CL approaches in terms of accurate inference, thereby offering a viable and robust solution for DL tasks. TL outperformed other DL methods and improved accuracy by 7.85% for independent and identically distributed (IID) datasets, macro F1-score by 1.06% for non-IID datasets, accuracy by 2.60% for text classification, and AUC by 3.88% and 4.54% for medical and financial datasets, respectively. By effectively preserving data privacy while maintaining performance, TL represents a significant advancement in DL methodologies.

Problem

• Addresses accuracy drop in distributed learning paradigms
• Improves gradient updates in split network models
• Enhances performance while preserving data privacy

Innovation

• Traversal Learning combines FP and BP in distributed nodes
• Orchestrator plans model visits and virtual batches
• TL matches centralized learning accuracy in distributed settings