SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

2026-08-13

AI Summary

This work addresses the challenge of deploying large language models (LLMs) at the edge, where high computational costs and accuracy degradation present a critical trade-off. The authors propose the first speculative decoding framework tailored for edge–cloud collaborative inference: a compact model on the edge rapidly generates draft tokens, which are then verified in parallel by a larger cloud-based model, with corrections applied only when predictions are incorrect. This approach drastically reduces the number of cloud model invocations without requiring any retraining, achieving zero accuracy loss. Evaluated across multiple natural language processing tasks, the method reduces cloud model calls by up to 76%, effectively balancing computational efficiency and prediction accuracy.

Abstract

Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, \our{}, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that \our{} reduces the cloud model calls by $76\%$ with zero loss in accuracy as compared to the full model.

Problem

• Large Language Models
• Edge-Cloud Inference
• Speculative Decoding
• Computational Cost
• Model Accuracy

Innovation

• Speculative Decoding
• Edge-Cloud Inference
• Distributed LLM Deployment
• Cost-Efficient Inference
• Plug-and-Play Framework