ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling

2026-08-08

AI Summary

This work addresses the challenge of fine-tuning large language models on edge devices, where backpropagation induces update locking, resulting in high memory overhead and low throughput. To overcome this limitation, the authors propose ZeroLock, a backpropagation-free training algorithm that decouples parameter updates by constructing local objective functions for generic model blocks, enabling concurrent execution of independent modules. The paper establishes the first theoretical framework for such local objectives, proving a convergence rate of Õ(1/√T)—only slower than backpropagation by a polylogarithmic factor. The system implementation integrates early forward computation with a fault-recovery mechanism. Experimental results demonstrate that ZeroLock reduces memory consumption by 26.5% and improves throughput by 4.9% compared to baseline methods.

Abstract

Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources of edge devices, they commonly rely on backpropagation (BP) training, which has a fundamental limitation of update locking and could experience severe throughput and memory bottlenecks. In this work, we propose a BP-free algorithm, called ZeroLock, that decouples the model updates into independent chunk updates by local objective construction. It breaks the update locking of BP and hence can improve throughput at the algorithm level and lower memory usage by reducing activation storage. To the best of our knowledge, we provide the first theoretical framework for such local objective construction-based approaches under general model chunk division by mapping local objectives to the global objective. We prove that ZeroLock has a convergence rate of $\tilde{\mathcal{O}}(1/\sqrt{T})$, which differs from BP only by polylogarithmic factors. We design a system for ZeroLock and build real-world prototypes, incorporating techniques such as early forwarding and failure recovery for efficient and robust implementation. Experiments on the prototype show that compared to BP-based baselines, ZeroLock reduces the memory by 26.5% and improves throughput by 4.9%.

Problem

• large language model
• edge fine-tuning
• update locking
• memory bottleneck
• throughput

Innovation

• modular update decoupling
• backpropagation-free training
• local objective construction
• memory-efficient LLM training
• convergence analysis