SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving

2025-06-11

AI Summary

To address the challenge of efficient large language model (LLM) inference under stringent memory and power constraints on resource-constrained edge devices, this paper proposes SLED, a collaborative speculative decoding framework. SLED pioneers the adaptation of speculative decoding to heterogeneous edge computing: lightweight client devices locally generate multiple draft tokens, while a shared edge server performs unified verification via dynamic batching—enabling hierarchical client-edge collaboration. It supports multi-client reuse of a single target model, significantly reducing GPU memory overhead on the server and ensuring compatibility with diverse hardware platforms. Evaluated on Jetson Orin Nano, Raspberry Pi 5, and RTX 6000, SLED achieves 32–57% end-to-end latency reduction and 2.1–3.8× energy efficiency improvement over baseline methods, while preserving original model accuracy; concurrent session capacity increases by 3.4×.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Regardless the advancements in device capabilities, efficient inferencing advanced large language models (LLMs) at the edge remains challenging due to limited device memory and power constraints. Existing strategies, such as aggressive quantization, pruning, or remote inference, trade accuracy for efficiency or lead to substantial cost burdens. This position paper introduces a new approach that leverages speculative decoding, previously viewed primarily as a decoding acceleration technique for autoregressive generation of LLMs, as a promising approach specifically adapted for edge computing by orchestrating computation across heterogeneous devices. We propose SLED, a method that allows lightweight edge devices to draft multiple candidate tokens locally using diverse draft models, while a single, shared edge server efficiently batches and verifies the tokens utilizing a more precise target model. This approach supports device heterogeneity and reduces server-side memory footprint by avoiding the need to deploy multiple target models. Our initial experiments with Jetson Orin Nano, Raspberry Pi 5, and an RTX 6000 edge server indicate substantial benefits: significantly reduced latency, improved energy efficiency, and increased concurrent inference sessions, all without sacrificing model accuracy.

Problem

• Efficient LLM inference at edge with limited resources
• Balancing accuracy and efficiency in edge computing
• Reducing latency and energy use in edge LLM serving

Innovation

• Speculative decoding for edge LLM serving
• Heterogeneous device computation orchestration
• Shared server verifies multiple local drafts