Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference

2024-12-25

AI Summary

To address the hardware heterogeneity of consumer-grade devices—characterized by relatively weak GPUs and powerful CPUs—this paper proposes a CPU/GPU-coordinated speculative decoding framework: a lightweight draft model runs on the GPU to generate candidate tokens, while the target model performs parallel verification on the CPU, substantially reducing cross-device communication overhead. Key contributions include: (1) the first CPU-GPU division-of-labor paradigm for speculative decoding; (2) a depth-enhanced lightweight draft model; (3) a Dynamic Gating Fusion (DGF) mechanism to improve efficiency in feature-embedding fusion; and (4) adaptive optimization of candidate token count to balance latency and throughput. On HumanEval, LLaMA2-Chat-7B achieves 5.86 tokens/s using only 3 GB VRAM—2.77× faster than CPU-only inference—and reaches 8 tokens/s with 7 GB VRAM, significantly enhancing on-device large language model inference efficiency.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Due to the high resource demands of Large Language Models (LLMs), achieving widespread deployment on consumer-grade devices presents significant challenges. Typically, personal or consumer-grade devices, including servers configured prior to the era of large-scale models, generally have relatively weak GPUs and relatively strong CPUs. However, most current methods primarily depend on GPUs for computation. Therefore, we propose Dovetail, an approach that deploys the draft model on the GPU to generate draft tokens while allowing the target model to perform parallel verification on the CPU, thereby improving the utilization of all available hardware resources and occupying less inter-device communication bandwidth. Accordingly, we have redesigned the draft model to better align with heterogeneous hardware characteristics. To this end, we implemented several optimizations: reducing the number of draft tokens to mitigate latency in parallel verification, increasing the depth of the draft model to enhance its predictive capacity, and introducing DGF (Dynamic Gating Fusion) to improve the integration of features and token embeddings. In the HumanEval benchmark, Dovetail achieved an inference speed of 5.86 tokens per second for LLaMA2-Chat-7B using 3GB of VRAM, representing an approximately 2.77x improvement over CPU-only inference. Furthermore, the inference speed was increased to 8 tokens per second when utilizing 7GB of VRAM.

Problem

• Large Language Models
• Resource-constrained Devices
• CPU-GPU Synergy

Innovation

• Dovetail Technology
• Resource Optimization
• Efficient Inference