WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference

2026-04-19

AI Summary

This work addresses the inefficiency of device-edge speculative decoding under wireless channel fluctuations, where conventional token-level matching verification often leads to excessive rejection, reduced acceptance length, and increased interaction rounds. To overcome these limitations, we propose WISV, a novel framework that integrates real-time channel state information (CSI) into semantic verification, thereby relaxing strict token alignment constraints. WISV employs a lightweight decision head that fuses high-dimensional hidden representations from large language models with CSI to dynamically evaluate speculated tokens. Additionally, it introduces a full-hidden upload strategy and a mismatch-priority selective upload protocol to balance verification accuracy against communication overhead. Experiments with a 1B draft model and an 8B target model demonstrate a 60.8% increase in acceptance length, a 37.3% reduction in interaction rounds, and a 31.4% decrease in end-to-end latency, with less than 1% degradation in task accuracy, validated on Jetson AGX Orin and A40 platforms.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

While distributed device-edge speculative decoding enhances resource utilization across heterogeneous nodes, its performance is often bottlenecked by conventional token-level verification strategies. Such rigid alignment leads to excessive rejections, significantly diminishing the accepted sequence length and increasing interaction rounds under fluctuating wireless conditions. In this paper, we propose WISV (Wireless-Informed Semantic Verification), a novel distributed speculative decoding framework that goes beyond strict token-level matching via a channel-aware semantic acceptance policy. WISV integrates a lightweight decision head into the edge-side target LLM to dynamically evaluate speculative tokens by synthesizing high-dimensional hidden representations with instantaneous channel state information (CSI). To optimize the trade-off between verification fidelity and communication overhead, we further design two tailored communication protocols: full-hidden upload and mismatch-first selective-hidden upload. Extensive simulations using a 1B drafter and an 8B target model demonstrate that WISV achieves up to a 60.8% increase in accepted length, a 37.3% reduction in interaction rounds, and a 31.4% improvement in end-to-end latency compared to vanilla speculative decoding across tested settings, while maintaining a negligible task accuracy drop (<1%). Finally, we validate WISV on a hardware testbed comprising an NVIDIA Jetson AGX Orin and an A40-equipped server, confirming its real-world efficacy in accelerating edge-deployed LLM inference.

Problem

• speculative decoding
• semantic verification
• wireless channel
• edge LLM inference
• token-level verification

Innovation

• semantic verification
• speculative decoding
• wireless-informed
• edge LLM inference
• channel state information