Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning

2026-01-14

AI Summary

This work addresses the high memory and computational overhead incurred by client-side backpropagation in Split Federated Learning, which limits the deployable model scale. To overcome this challenge, the authors propose HERON-SFL, a novel framework that uniquely integrates zeroth-order optimization on the client with first-order optimization on the server. Clients employ a lightweight auxiliary network to approximate gradients via forward-only perturbation evaluations, thereby eliminating explicit gradient computation and activation caching. Theoretical analysis demonstrates that the convergence rate of HERON-SFL is independent of model dimensionality, effectively circumventing the scalability bottleneck inherent in conventional zeroth-order methods. Empirical results show that HERON-SFL matches baseline accuracy in both ResNet training and language model fine-tuning tasks while reducing peak client memory usage by up to 64% and per-iteration computational cost by up to 33%.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Split Federated Learning (SFL) enables collaborative training between resource-constrained edge devices and a compute-rich server. Communication overhead is a central issue in SFL and can be mitigated with auxiliary networks. Yet, the fundamental client-side computation challenge remains, as back-propagation requires substantial memory and computation costs, severely limiting the scale of models that edge devices can support. To enable more resource-efficient client computation and reduce the client-server communication, we propose HERON-SFL, a novel hybrid optimization framework that integrates zeroth-order (ZO) optimization for local client training while retaining first-order (FO) optimization on the server. With the assistance of auxiliary networks, ZO updates enable clients to approximate local gradients using perturbed forward-only evaluations per step, eliminating memory-intensive activation caching and avoiding explicit gradient computation in the traditional training process. Leveraging the low effective rank assumption, we theoretically prove that HERON-SFL's convergence rate is independent of model dimensionality, addressing a key scalability concern common to ZO algorithms. Empirically, on ResNet training and language model (LM) fine-tuning tasks, HERON-SFL matches benchmark accuracy while reducing client peak memory by up to 64% and client-side compute cost by up to 33% per step, substantially expanding the range of models that can be trained or adapted on resource-limited devices.

Problem

• Split Federated Learning
• resource-constrained devices
• communication overhead
• client-side computation
• memory efficiency

Innovation

• Split Federated Learning
• Zeroth-Order Optimization
• Hybrid Optimization
• Resource-Efficient Training
• Convergence Analysis