Adaptive Matrix Multiplication for Dynamic Shapes on Ascend NPUs

2026-08-11

AI Summary

This work addresses the challenge of efficiently executing matrix multiplication with dynamic tensor shapes on Ascend NPUs, where conventional GPU-oriented optimization techniques fail to deliver performance. To overcome this, the authors propose AdaptCore, a novel framework that, for the first time, decouples dynamic GEMM optimization into two orthogonal dimensions: spatial tiling and instruction scheduling. By integrating hardware-aware 2D tiling classification, a composable optimization library, and a deterministic performance model, AdaptCore enables O(1) runtime scheduling and cache management overhead. Evaluated across 80,000 diverse dynamic input shapes, AdaptCore achieves an average speedup of 1.85×, with up to 1.48× end-to-end acceleration on representative models, significantly outperforming the native ACLNN library.

Abstract

Matrix Multiplication (MatMul) faces a "generalization crisis" driven by highly dynamic tensor shapes. This crisis is particularly acute on Ascend NPUs, where explicitly controlled architectures and strict physical constraints render existing GPU-centric optimizations ineffective. To resolve this, we propose AdaptCore, an adaptive framework for universally high-performance MatMul on Ascend NPUs. AdaptCore systematically decouples operator optimization into spatial tiling and instruction orchestration. It first maps dynamic shapes into a hardware-aware 2D tiling taxonomy to balance on-chip capacity limits and multi-core parallelism. Furthermore, it integrates a composable optimization library with a deterministic analytical performance model. By mathematically evaluating hardware state mutations, AdaptCore proactively selects and caches optimal implementations, enabling O(1) overhead runtime dispatching. Evaluations demonstrate that AdaptCore delivers a remarkable 1.85x mean speedup across 80,000 input shapes, and achieves up to a 1.48x acceleration in representative end-to-end models over the highly-tuned native vendor library (ACLNN).

Problem

• Matrix Multiplication
• Dynamic Shapes
• Ascend NPUs
• Generalization Crisis
• Hardware Constraints

Innovation

• adaptive matrix multiplication
• dynamic tensor shapes
• Ascend NPU
• spatial tiling
• instruction orchestration