Phase Matters: Characterizing Heterogeneous Vision-Language Inference on a Mobile SoC

2026-06-26

AI Summary

This work addresses the lack of fine-grained deployment guidance for mobile visual language models (VLMs) on heterogeneous hardware such as CPUs and NPUs, which hinders efficient utilization of acceleration units. Conducting hardware-in-the-loop experiments on a Qualcomm SM8750 platform, the study reveals for the first time the “phase effect” between prefill and decode stages in VLM inference and systematically evaluates the acceleration potential of image encoders across different architectures. Based on these insights, the authors propose a four-step graph rewriting strategy that successfully integrates previously unsupported encoders—such as Phi-3.5-V—into the Qualcomm Neural Network (QNN) execution path. Experiments demonstrate that NPU acceleration achieves speedups of 1.64× and 1.18× in the prefill and decode phases, respectively, while visual encoders attain 20–45× acceleration, reducing system steady-state temperature by 10.47°C and energy consumption by 2.52×, culminating in an end-to-end 22× speedup for Phi-3.5-V.

Abstract

Recent phone-class mobile SoCs expose practical NPU execution paths for on-device vision-language model (VLM) inference, but developers still lack phase-level guidance for mapping VLM pipelines across heterogeneous backends. We present a hardware-in-the-loop characterization of VLM inference on the Qualcomm SM8750 (Snapdragon 8 Elite), covering phase throughput, cache-state effects, 100-run thermal stability, energy, heterogeneous CPU/NPU pipeline configurations, and visual-token-budget sensitivity. Using FastVLM-0.5B as an end-to-end case study, together with encoder-only measurements across four architecture families, we show that phase matters: NPU execution is highly phase-dependent, delivering 1.64x speedup for prefill but only 1.18x for decode, while vision encoders achieve 20-45x speedups over CPU. These gains translate into 10.47 degrees C lower steady-state temperature and 2.52x lower energy, avoiding thermal throttling in always-on settings. Finally, we show that a four-step graph rewrite enables previously unsupported encoders, such as Phi-3.5-V, to reach the QNN path with up to 22x speedup, providing a practical porting recipe for mobile VLM deployment.

Problem

• vision-language models
• heterogeneous inference
• mobile SoC
• phase-level characterization
• on-device AI

Innovation

• heterogeneous inference
• phase-level characterization
• mobile VLM
• NPU acceleration
• graph rewriting