A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference
🤖 AI Summary
This work addresses the high latency and substantial cost of large language model inference in bandwidth-constrained edge–cloud collaborative settings, where computational load imbalance further exacerbates performance bottlenecks. To tackle these challenges, the authors propose PicoSpec, a novel framework featuring the first asynchronous pipelined speculative decoding architecture tailored for edge–cloud scenarios. PicoSpec integrates sparse compression with an independent rejection sampling mechanism to enable training-free, efficient collaborative inference. Its key innovation lies in eliminating mutual waiting between edge and cloud components and performing rejection sampling via a single transmission of a compressed vocabulary, drastically reducing communication overhead. Experimental results demonstrate that PicoSpec achieves up to 2.9× speedup over state-of-the-art methods while significantly lowering communication latency and resource consumption.
Application Category
📝 Abstract
Recent advancements and widespread adoption of Large Language Models (LLMs) in both industry and academia have catalyzed significant demand for LLM serving. However, traditional cloud services incur high costs, while on-device inference alone faces challenges due to limited resources. Edge-cloud collaboration emerges as a key research direction to combine the strengths of both paradigms, yet efficiently utilizing limited network bandwidth while fully leveraging and balancing the computational capabilities of edge devices and the cloud remains an open problem. To address these challenges, we propose Pipelined Collaborative Speculative Decoding Framework (PicoSpec), a novel, general-purpose, and training-free speculative decoding framework for LLM edge-cloud collaborative inference. We design an asynchronous pipeline that resolves the mutual waiting problem inherent in vanilla speculative decoding within edge collaboration scenarios, which concurrently executes a Small Language Model (SLM) on the edge device and a LLM in the cloud. Meanwhile, to mitigate the significant communication latency caused by transmitting vocabulary distributions, we introduce separate rejection sampling with sparse compression, which completes the rejection sampling with only a one-time cost of transmitting the compressed vocabulary. Experimental results demonstrate that our solution outperforms baseline and existing methods, achieving up to 2.9 speedup.
Problem
Research questions and friction points this paper is trying to address.
edge-cloud collaboration
large language models
speculative decoding
network bandwidth
computational resource balancing
Innovation
Methods, ideas, or system contributions that make the work stand out.
speculative decoding
edge-cloud collaboration
asynchronous pipeline
rejection sampling
sparse compression
🔎 Similar Papers
No similar papers found.
