On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend

2026-07-09

AI Summary

This work addresses the significant engineering and platform-support challenges faced by non-GPU AI accelerators in deploying Mixture-of-Experts (MoE) and multimodal large language models. For the first time, it systematically identifies eight major categories of limitations inherent to such hardware during large-model inference. Leveraging Huawei’s Ascend 910 accelerator with the CANN software stack and a customized vLLM-Ascend framework, the authors successfully deploy the DeepSeek-V4-Flash family of MoE and vision-language models through W8A8 quantization, source-level plugin patches, and runtime fault-tolerance mechanisms. Experimental results demonstrate stable, high-concurrency multimodal inference for medical applications, achieving reliable performance on MMMU/MMMU-Pro benchmarks and safety alignment evaluations, while also quantifying the associated integration overhead and quality-of-service trade-offs.

Abstract

Non-GPU AI accelerators are increasingly adopted as alternatives to general-purpose GPUs for large-model inference, but the real engineering cost of migrating demanding workloads beyond CUDA remains poorly documented. We present a field study of deploying two large inference workloads on a 16-device Huawei Ascend 910 system using CANN and vLLM-Ascend: an LLM-as-a-judge safety and alignment evaluation pipeline based on a W8A8 MoE judge model, DeepSeek-V4-Flash, and a multimodal medical vision--language benchmark based on DeepSeek-V4-Flash-Vision for MMMU and MMMU-Pro. Making these workloads reliable required twelve source-level patches to the vendor inference plugin, disabling several high-throughput features to preserve numerical correctness, and adding operational safeguards for recurring device-level failures. We summarize the main platform limitations in eight categories: incomplete operator and feature support, fragile parallelism, numerical faults in low-level kernels, immature graph compilation, unstable advanced features, limited scalability, weak observability, and ecosystem fragmentation. For each category, we report the symptoms, evidence, and likely causes. We also quantify the integration effort, concurrency behavior, and benchmark quality to show that both workloads were served correctly. Our study provides a reproducible reference for teams evaluating or operating non-GPU accelerators for large-model inference.

Problem

• non-GPU AI accelerators
• large-model inference
• MoE
• multimodal serving
• platform limitations

Innovation

• non-GPU accelerators
• large-model inference
• MoE
• multimodal serving
• Ascend