Optimizing Structured-Sparse Matrix Multiplication in RISC-V Vector Processors

2025-01-17

AI Summary

To address low computational efficiency of sparse matrix multiplication—particularly for structured-sparse CNN inference—on RISC-V vector processors, this paper proposes a hardware-software co-design methodology. First, it analyzes sparse data distribution and memory locality characteristics. Based on this analysis, it introduces a lightweight hardware extension: the novel *vindexmac* instruction, the first to support indirect addressing of vector registers, enabling efficient fusion of sparse index lookup and multiply-accumulate operations. The approach further integrates RISC-V Vector Extension (V) features, structured-sparse encoding, register-level data layout optimization, and microarchitectural co-design. Evaluated on mainstream CNN models, the solution achieves 25–33% runtime speedup over highly optimized vectorized kernels, with negligible hardware overhead. This significantly improves both execution efficiency and scalability of sparse CNN workloads on RISC-V vector platforms.

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Responsible Web: Measurement, analysis, and circumvention of Web censorship

Abstract

Structured sparsity has been proposed as an efficient way to prune the complexity of Machine Learning (ML) applications and to simplify the handling of sparse data in hardware. Accelerating ML models, whether for training, or inference, heavily relies on matrix multiplications that can be efficiently executed on vector processors, or custom matrix engines. This work aims to integrate the simplicity of structured sparsity into vector execution to speed up the corresponding matrix multiplications. Initially, the implementation of structured-sparse matrix multiplication using the current RISC-V instruction set vector extension is comprehensively explored. Critical parameters that affect performance, such as the impact of data distribution across the scalar and vector register files, data locality, and the effectiveness of loop unrolling are analyzed both qualitatively and quantitatively. Furthermore, it is demonstrated that the addition of a single new instruction would reap even higher performance. The newly proposed instruction is called vindexmac, i.e., vector index-multiply-accumulate. It allows for indirect reads from the vector register file and it reduces the number of instructions executed per matrix multiplication iteration, without introducing additional dependencies that would limit loop unrolling. The proposed new instruction was integrated in a decoupled RISC-V vector processor with negligible hardware cost. Experimental results demonstrate the runtime efficiency and the scalability offered by the introduced optimizations and the new instruction for the execution of state-of-the-art Convolutional Neural Networks. More particularly, the addition of a custom instruction improves runtime by 25% and 33% when compared with highly-optimized vectorized kernels that use only the currently defined RISC-V instructions.

Problem

• RISC-V
• Sparse Matrix Multiplication
• Machine Learning Optimization

Innovation

• vindexmac
• RISC-V
• structured sparsity optimization