AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices

2026-04-28

AI Summary

This work addresses the significant synchronization-induced idle overhead in mobile single-NPU-PIM systems and the computational waste caused by asynchronous draft-length fluctuations in speculative decoding. To mitigate these issues, the authors propose a task-level asynchronous NPU-PIM heterogeneous architecture that decouples draft generation (DLM) and verification (TLM), enabling PIM to generate drafts in parallel while the NPU performs synchronous verification. The design incorporates entropy- and history-aware adaptive draft control alongside time-aware pre-verification scheduling to effectively suppress low-confidence, invalid drafts. Integrated within an LPDDR5-PIM substrate, the system features dedicated attention algorithm units and a gated task scheduler, supporting localized attention computation and sub-microsecond task switching. Compared to a GPU-only baseline, the proposed architecture achieves 4.2× higher throughput and 5.6× better energy efficiency, outperforming existing GPU+PIM approaches by 1.5× in throughput and 1.24× in energy efficiency, with hardware overhead under 3% of DRAM area.

Abstract

Speculative decoding enhances the inference efficiency of large language models (LLMs) by generating drafts using a small draft language model (DLM) and verifying them in batches with a large target language model (TLM). However, adaptive drafting inference on a mobile single-NPU-PIM system faces idle overhead in traditional operator-level synchronous execution and wasted computation in asynchronous execution due to fluctuations in draft length. This paper introduces AHASD, a task-level asynchronous mobile NPU-PIM heterogeneous architecture for speculative decoding. Notably, AHASD achieves parallel drafting on the PIM and verification on a single NPU through task-level DLM-TLM decoupling and specifically, it incorporates Entropy-History-Aware Drafting Control and Time-Aware Pre-Verification Control to dynamically manage adaptive drafting algorithm execution and pre-verification timing, suppressing invalid drafting based on low-confidence drafts. Additionally, AHASD integrates Attention Algorithm Units and Gated Task Scheduling Units within LPDDR5-PIM to enable attention link localization and sub-microsecond task switching on the PIM side. Experimental results for different LLMs and adaptive drafting algorithms show that AHASD achieves up to 4.2$\times$ in throughput and 5.6$\times$ in energy efficiency improvements over a GPU-only baseline, and 1.5$\times$ in throughput and 1.24$\times$ in energy efficiency gains over the state-of-the-art GPU+PIM baseline, with hardware overhead below 3\% of the DRAM area.

Problem

• speculative decoding
• adaptive drafting
• mobile NPU-PIM
• idle overhead
• wasted computation

Innovation

• speculative decoding
• NPU-PIM heterogeneous architecture
• adaptive drafting
• task-level asynchrony
• entropy-history-aware control