GAPSL: A Gradient-Aligned Parallel Split Learning on Heterogeneous Data

2026-03-19

AI Summary

This work addresses the challenge of training divergence in Parallel Split Learning (PSL) under heterogeneous data, which arises from inconsistent gradient directions across clients and impedes model convergence. To mitigate this issue, the authors propose GAPSL, a novel framework that introduces a Leader Gradient Identification (LGI) mechanism to dynamically select clients with aligned gradient directions as leaders. GAPSL further incorporates Gradient Direction Alignment (GDA) and direction-aware regularization to effectively alleviate cross-device gradient conflicts—without requiring global model aggregation. Experimental results on a prototype platform demonstrate that GAPSL significantly outperforms existing methods, achieving higher training accuracy while reducing communication latency.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The increasing complexity of neural networks poses significant challenges for democratizing FL on resource?constrained client devices. Parallel split learning (PSL) has emerged as a promising solution by offloading substantial computing workload to a server via model partitioning, shrinking client-side computing load, and eliminating the client-side model aggregation for reduced communication and deployment costs. Since PSL is aggregation-free, it suffers from severe training divergence stemming from gradient directional inconsistency across clients. To address this challenge, we propose GAPSL, a gradient-aligned PSL framework that comprises two key components: leader gradient identification (LGI) and gradient direction alignment (GDA). LGI dynamically selects a set of directionally consistent client gradients to construct a leader gradient that captures the global convergence trend. GDA employs a direction-aware regularization to align each client's gradient with the leader gradient, thereby mitigating inter-device gradient directional inconsistency and enhancing model convergence. We evaluate GAPSL on a prototype computing testbed. Extensive experiments demonstrate that GAPSL consistently outperforms state-of-the-art benchmarks in training accuracy and latency.

Problem

• Parallel Split Learning
• Gradient Directional Inconsistency
• Training Divergence
• Heterogeneous Data
• Federated Learning

Innovation

• Gradient Alignment
• Parallel Split Learning
• Heterogeneous Data
• Leader Gradient
• Direction-aware Regularization