Multi-objective Differentiable Neural Architecture Search

2024-02-28

arXiv.org

AI Summary

Multi-objective neural architecture search (MOO-NAS) for heterogeneous devices suffers from low Pareto-frontier construction efficiency—especially when jointly optimizing accuracy and diverse hardware metrics (e.g., latency, energy)—as conventional constraint-driven methods require multiple independent searches, incurring prohibitive computational overhead. Method: We propose the first hardware-aware supernet framework enabling single-shot differentiable MOO-NAS. It jointly models architectural distributions via preference-vector guidance and hardware-feature conditional modeling, allowing direct generation of diverse Pareto-optimal architectures for multiple devices within one training run, achieving zero-shot cross-device transfer. Results: Our method achieves state-of-the-art performance across 19 hardware platforms, three task domains (e.g., image classification on ImageNet-1K, machine translation), and heterogeneous search spaces—outperforming all existing MOO-NAS approaches in both Pareto-frontier quality and search efficiency.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training a neural network. Typically, in MOO for neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments involving up to 19 hardware devices and 3 different objectives demonstrate the effectiveness and scalability of our method. Finally, we show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.

Problem

• Balancing performance and hardware metrics across devices
• Profiling Pareto front in multi-objective optimization efficiently
• Enabling zero-shot transferability to new hardware devices

Innovation

• Hypernetwork parameterizes architectural distribution
• Encodes user preferences for trade-offs
• Enables zero-shot transfer to new devices