Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits

2025-05-27

AI Summary

To address the challenges of high cloud-API dependency and insufficient on-device computational resources—hindering simultaneous achievement of accuracy and efficiency when deploying large language models (LLMs) on edge devices—this paper proposes a cloud-edge collaborative speculative decoding framework. A lightweight draft model (Vicuna-68M) runs locally on the edge, while a large target model (Llama2-7B) resides in the cloud. We introduce, for the first time, an early-exit mechanism into speculative decoding: the target model verifies the current token while concurrently exiting early to generate subsequent candidate tokens, enabling asynchronous client-side pre-generation and token-level verification in parallel. This design significantly improves cloud-edge resource utilization and throughput. Experiments show a 35% latency reduction on the Jetson Nano + A100 platform; integrating pre-generation further accelerates inference by 11%. In a vision-language-model (VLM)-driven control task on the Unitree Go2 robot, end-to-end latency decreases by 21%.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Systems and Infrastructure for Web, Mobile and WoT: Cloud, edge and content delivery systems for the Web

Abstract

Large Language Models (LLMs) enable various applications on edge devices such as smartphones, wearables, and embodied robots. However, their deployment often depends on expensive cloud-based APIs, creating high operational costs, which limit access for smaller organizations and raise sustainability concerns. Certain LLMs can be deployed on-device, offering a cost-effective solution with reduced latency and improved privacy. Yet, limited computing resources constrain the size and accuracy of models that can be deployed, necessitating a collaborative design between edge and cloud. We propose a fast and cost-effective speculative edge-cloud decoding framework with a large target model on the server and a small draft model on the device. By introducing early exits in the target model, tokens are generated mid-verification, allowing the client to preemptively draft subsequent tokens before final verification, thus utilizing idle time and enhancing parallelism between edge and cloud. Using an NVIDIA Jetson Nano (client) and an A100 GPU (server) with Vicuna-68M (draft) and Llama2-7B (target) models, our method achieves up to a 35% reduction in latency compared to cloud-based autoregressive decoding, with an additional 11% improvement from preemptive drafting. To demonstrate real-world applicability, we deploy our method on the Unitree Go2 quadruped robot using Vision-Language Model (VLM) based control, achieving a 21% speedup over traditional cloud-based autoregressive decoding. These results demonstrate the potential of our framework for real-time LLM and VLM applications on resource-constrained edge devices.

Problem

• High cloud API costs limit LLM access for small organizations
• On-device LLMs face constraints in model size and accuracy
• Edge-cloud collaboration needed for efficient LLM deployment

Innovation

• Speculative edge-cloud decoding with early exits
• Preemptive drafting utilizing idle time
• Combining small draft and large target models