Leveraging AutoML for Sustainable Deep Learning: A Multi-Objective HPO Approach on Deep Shift Neural Networks

2026-06-22

AI Summary

This work addresses the substantial environmental burden imposed by the high computational cost of deep learning models by introducing, for the first time, a multi-objective AutoML framework tailored for deep shift neural networks (DSNNs)—an energy-efficient alternative that has lacked systematic optimization. The proposed approach integrates multi-fidelity Bayesian optimization with mixed-precision quantization to automatically discover Pareto-optimal trade-offs between accuracy and energy consumption in image classification tasks. Notably, the study uncovers counterintuitive yet highly efficient low-precision configurations, achieving approximately 20% performance improvement and reducing carbon emissions by over 60%. The method’s effectiveness and generalizability are further validated across multiple backbone architectures, demonstrating its broad applicability in sustainable deep learning design.

Abstract

Deep Learning (DL) has advanced various fields by extracting complex patterns from large datasets. However, the computational demands of DL models pose environmental and resource challenges. Deep Shift Neural Networks (DSNNs) present a solution by leveraging shift operations to reduce computational complexity at inference. Compared to common DNNs, DSNNs are still less well understood and less well optimized. By leveraging AutoML techniques, we provide valuable insights into the potential of DSNNs and how to design them in a better way. We focus on image classification, a core task in computer vision, especially in low-resource environments. Since we consider complementary objectives such as accuracy and energy consumption, we combine state-of-the-art multi-fidelity (MF) hyperparameter optimization (HPO) with multi-objective optimization to find a set of Pareto optimal trade-offs on how to design DSNNs. Our approach led to significantly better configurations of DSNNs regarding loss and emissions compared to default DSNNs. This includes simultaneously increasing performance by about 20% and reducing emissions, in some cases by more than 60%. Investigating the behavior of quantized networks in terms of both emissions and accuracy, our experiments reveal surprising model-specific trade-offs, yielding the greatest energy savings. For example, in contrast to common expectations, quantizing smaller portions of the network with low precision can be optimal with respect to energy consumption while retaining or improving performance. We corroborated these findings across multiple backbone architectures, highlighting important nuances in quantization strategies and offering an automated approach to balancing energy efficiency and model performance.

Problem

• Deep Shift Neural Networks
• Sustainable Deep Learning
• Multi-Objective Optimization
• Hyperparameter Optimization
• Energy Efficiency

Innovation

• AutoML
• Multi-objective Optimization
• Deep Shift Neural Networks
• Hyperparameter Optimization
• Energy-efficient Deep Learning