NPUsper: Eliminating Redundant Computation for Real-Time Whisper on Mobile NPUs

2026-07-01

AI Summary

This work addresses the redundant computation and high latency inherent in existing streaming Whisper inference on mobile NPUs. The authors propose an online hallucination token detection mechanism based on cross-attention patterns, eliminating the extensive padding required by conventional streaming systems. Furthermore, they introduce an NPU-optimized, controllable unrolling strategy that reformulates autoregressive decoding into K-step block-graph execution, substantially reducing KV cache computation and scheduling overhead. This approach achieves, for the first time, low-latency and low-power real-time Whisper inference on mobile NPUs while preserving transcription accuracy—yielding up to a 4.84× reduction in word-level latency, a 33.2× decrease in time-to-first-token (TTFT), and an average power consumption reduction of 88.64%.

Abstract

We present NPUsper, a live transcription system that makes Whisper efficient on mobile NPUs by eliminating redundant computation. To avoid the heavy padding used by prior streaming systems, NPUsper detects hallucinated tokens online from temporal patterns in decoder cross-attention, allowing each inference round to process short audio inputs with minimal carryover. For efficient mobile-NPU execution, we propose controlled unrolling, which executes autoregressive decoding as K-step chunk graphs, removing unnecessary KV-cache computation and reducing graph-dispatch overhead. NPUsper achieves up to 4.84x lower per-word latency, up to 33.2x lower time-to-first-token (TTFT), and up to 88.64% lower average power consumption compared with baselines, while maintaining comparable transcription accuracy. The code is available at https://github.com/npusper/NPUsper.

Problem

• redundant computation
• real-time Whisper
• mobile NPUs
• streaming transcription
• latency reduction

Innovation

• redundant computation elimination
• hallucination detection
• controlled unrolling
• mobile NPU optimization
• streaming Whisper