Efficient Global Neural Architecture Search

2025-02-05

AI Summary

Neural architecture search (NAS) suffers from high computational cost, manual design of macro-architectural configurations (e.g., depth and width), and unfairness in existing proxy-based performance estimation due to lack of architecture adaptivity. To address these issues, this paper proposes a globally navigable macro-micro joint search framework. We introduce the first macro-micro decoupled search paradigm, enabling fully automated co-optimization of depth and width via a hybrid search space. Furthermore, we design an architecture-aware dynamic training approximation mechanism that delivers low-overhead, differentiated performance prediction. Our method achieves state-of-the-art results on EMNIST and KMNIST, outperforms prior approaches on CIFAR-10, CIFAR-100, and Fashion-MNIST, accelerates search by 2–4× over the fastest global NAS methods, and successfully transfers to face recognition tasks.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neural architecture search (NAS) has shown promise towards automating neural network design for a given task, but it is computationally demanding due to training costs associated with evaluating a large number of architectures to find the optimal one. To speed up NAS, recent works limit the search to network building blocks (modular search) instead of searching the entire architecture (global search), approximate candidates' performance evaluation in lieu of complete training, and use gradient descent rather than naturally suitable discrete optimization approaches. However, modular search does not determine network's macro architecture i.e. depth and width, demanding manual trial and error post-search, hence lacking automation. In this work, we revisit NAS and design a navigable, yet architecturally diverse, macro-micro search space. In addition, to determine relative rankings of candidates, existing methods employ consistent approximations across entire search spaces, whereas different networks may not be fairly comparable under one training protocol. Hence, we propose an architecture-aware approximation with variable training schemes for different networks. Moreover, we develop an efficient search strategy by disjoining macro-micro network design that yields competitive architectures in terms of both accuracy and size. Our proposed framework achieves a new state-of-the-art on EMNIST and KMNIST, while being highly competitive on the CIFAR-10, CIFAR-100, and FashionMNIST datasets and being 2-4x faster than the fastest global search methods. Lastly, we demonstrate the transferability of our framework to real-world computer vision problems by discovering competitive architectures for face recognition applications.

Problem

• Automates global neural architecture search for efficiency
• Addresses unfair network comparisons in performance evaluations
• Enhances NAS with macro-micro search space design

Innovation

• Macro-micro search space design
• Architecture-aware approximation strategy
• Efficient disjoined search strategy