SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models

2025-03-20

AI Summary

Existing speculative decoding methods employ homogeneous small models, limiting adaptability to varying request complexities, exhibiting poor batch-processing support, and lacking co-optimization between speculation and verification stages. To address these limitations, we propose a heterogeneous speculative decoding system. First, we introduce a novel dynamic selection mechanism for multiple heterogeneous speculative models, adaptively assigning the optimal small model based on per-request difficulty estimation. Second, we design a request decomposition scheduling strategy to reduce batch overhead during verification. Third, we develop a GPU-level speculation-verification pipelined execution framework to tightly couple the two stages. Experimental results demonstrate a 2.28× speedup in end-to-end inference latency over state-of-the-art approaches, achieving superior throughput and latency trade-offs. Our work establishes a new paradigm for efficient large language model inference through principled heterogeneity and system-level co-design.

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Speculative decoding has been shown as an effective way to accelerate Large Language Model (LLM) inference by using a Small Speculative Model (SSM) to generate candidate tokens in a so-called speculation phase, which are subsequently verified by the LLM in a verification phase. However, current state-of-the-art speculative decoding approaches have three key limitations: handling requests with varying difficulty using homogeneous SSMs, lack of robust support for batch processing, and insufficient holistic optimization for both speculation and verification phases. In this paper, we introduce SPIN, an efficient LLM inference serving system based on speculative decoding, designed to address these challenges through three main innovations. First, SPIN improves token speculation by using multiple heterogeneous SSMs, with a learning-based algorithm for SSM selection that operates without prior knowledge of request difficulty. Second, SPIN employs a request decomposition method to minimize batching overhead during LLM verification. Finally, SPIN orchestrates speculation and verification phases by pipelining their executions on GPUs to achieve further acceleration. Experimental results demonstrate that SPIN significantly outperforms state-of-the-art methods, achieving a performance increase of approximately 2.28X.

Problem

• Handles varying request difficulty with heterogeneous SSMs
• Minimizes batching overhead during LLM verification
• Pipelines speculation and verification phases for acceleration

Innovation

• Uses multiple heterogeneous SSMs for token speculation
• Implements request decomposition to reduce batching overhead
• Pipelines speculation and verification phases on GPUs