Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization

2026-07-07

AI Summary

This work addresses the challenge of deploying deep neural networks on edge devices under stringent latency constraints while maintaining high accuracy—a trade-off inadequately handled by conventional approaches that lack direct control over inference delay. The authors propose a hardware-aware neural architecture learning method tailored for hard latency requirements, integrating a Zeroing Batch Normalization (ZeroBN) mechanism and a hardware-customized single-shot latency predictor. This enables the generation of efficient models meeting specific delay targets through only a single training run. By jointly optimizing hardware-aware latency modeling, one-shot neural architecture search, and compression with quantization, the approach achieves significant results: on a Jetson Nano, GoogLeNet’s latency is reduced to 34 ms with merely a 0.14% accuracy drop; on a Jetson TX2, VGG-19 and GoogLeNet attain accuracy gains of 0.5% and 0.78%, respectively, while operating within the same 34 ms latency budget.

Abstract

Deep learning applications have been widely adopted on edge devices, to mitigate the privacy and latency issues of accessing cloud servers. Deciding the number of neurons during the design of a deep neural network to maximize performance is not intuitive. Particularly, many application scenarios are real-time and have a strict latency constraint, while conventional neural network optimization methods do not directly change the temporal cost of model inference for latency-critical edge systems. In this work, we propose a latency-oriented neural network learning method to optimize models for high accuracy while fulfilling the latency constraint. For efficiency, we also introduce a universal hardware-customized latency predictor to optimize this procedure to learn a model that satisfies the latency constraint by only a one-shot training process. The experiment results reveal that, compared to state-of-the-art methods, our approach can well-fit the 'hard' latency constraint and achieve high accuracy. Under the same training settings as the original model and satisfying a 34 ms latency constraint on the ImageNet-100 dataset, we reduce GoogLeNet's latency from 40.32 ms to 34 ms with a 0.14% accuracy reduction on the NVIDIA Jetson Nano. When coupled with quantization, our method can be further improved to only 0.04% drop for GoogLeNet. On the NVIDIA Jetson TX2, we compress VGG-19 from 119.98 ms to 34 ms and even improve its accuracy by 0.5%, and we scale GoogLeNet up from 20.27 ms to 34 ms and achieve higher accuracy by 0.78%. We also open source this framework at https://github.com/ntuliuteam/ZeroBN

Problem

• latency constraint
• edge systems
• DNN architecture
• real-time inference
• neural network optimization

Innovation

• Latency-Constrained Architecture Search
• Zerorized Batch Normalization
• Edge AI
• One-Shot Neural Architecture Optimization
• Hardware-Aware Latency Prediction