Co-Design of CNN Accelerators for TinyML using Approximate Matrix Decomposition

2026-04-17

AI Summary

This work addresses the challenge of achieving low-latency, energy-efficient inference under stringent data privacy and hardware resource constraints in TinyML scenarios. To this end, the authors propose a training-free co-optimization framework that, for the first time, integrates genetic algorithm-driven approximate matrix decomposition into post-training quantization. This approach replaces multiplications in convolutional neural networks with shifts and additions, combined with power-of-two quantization, to jointly search for a multiplier-free FPGA accelerator architecture. Without requiring model retraining, the method enables hardware-friendly, accuracy-controllable compression, reducing average inference latency by 33% across multiple TinyML benchmarks while incurring only a 1.3% accuracy drop, thereby effectively meeting strict resource and precision requirements.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The paradigm shift towards local and on-device inference under stringent resource constraints is represented by the tiny machine learning (TinyML) domain. The primary goal of \gls{tml} is to integrate intelligence into tiny, low-cost devices under strict resource, energy, and latency constraints. However, the ultra-resource-constrained nature of these devices can lead to increased inference execution time, which can be detrimental in latency critical applications. At the same time, TinyML applications are often associated with sensitive data. As such, latency optimization approaches that rely on training samples are infeasible when such data is unavailable, proprietary, or sensitive, highlighting a pressing need for optimization approaches that do not require access to the training dataset and can be applied directly to pre-trained models. Replacing costly multiplications with more hardware-efficient operations, such as shifts and additions, has been proposed as an effective method for reducing inference latency. However, post-training power-of-two (Po2) approaches are scarce and, in many cases, lead to unacceptable accuracy loss. In this work, we propose a framework that applies approximate matrix decomposition to a given CNN in order to optimize hardware implementations subject to strict constraints and without any need of re-training or fine-tuning steps. The genetic algorithm-driven framework explores different matrix decompositions and resulting multiplier-less CNN accelerator designs for FPGA targets. A comprehensive evaluation of different TinyML benchmarks demonstrates our framework's efficacy in generating latency-optimized implementations that satisfy strict accuracy and resource constraints, achieving an average 33% latency improvement with an average accuracy loss of 1.3% compared to typical systolic array-based FPGA accelerators.

Problem

• TinyML
• CNN accelerator
• latency optimization
• post-training optimization
• resource-constrained devices

Innovation

• Approximate Matrix Decomposition
• Multiplier-less CNN
• Post-training Optimization
• TinyML Accelerator
• Genetic Algorithm