G-ICSO-NAS: Shifting Gears between Gradient and Swarm for Robust Neural Architecture Search

2026-04-01

AI Summary

This work addresses the tension in neural architecture search between the high computational cost of evolutionary algorithms and the premature convergence tendency of gradient-based methods by proposing a three-stage hybrid optimization framework. In the warm-up stage, the supernet weights are pre-trained with the architecture frozen. The exploration stage jointly updates architectures and weights via an improved competitive swarm optimizer (ICSO) integrated with gradient descent, where ICSO employs a diversity-aware fitness mechanism to enhance exploration. Finally, the stabilization stage refines the solution through fine-grained gradient-based search toward convergence, aided by dynamic switching and early-stopping strategies to reduce computational overhead. The method achieves 97.46% accuracy on CIFAR-10 with only 0.15 GPU-days, 83.1% on CIFAR-100, 75.02% on ImageNet, and sets new state-of-the-art results across all datasets in NAS-Bench-201.

Abstract

Neural Architecture Search (NAS) has become a pivotal technique in automated machine learning. Evolutionary Algorithm (EA)-based methods demonstrate superior search quality but suffer from prohibitive computational costs, while gradient-based approaches like DARTS offer high efficiency but are prone to premature convergence and performance collapse. To bridge this gap, we propose G-ICSO-NAS, a hybrid framework implementing a three-stage optimization strategy. The Warm-up Phase pre-trains supernet weights ($w$) via differentiable methods while architecture parameters ($α$) remain frozen. The Exploration Phase adopts a hybrid co-optimization mechanism: an Improved Competitive Swarm Optimizer (ICSO) with diversity-aware fitness navigates the architecture space to update $α$, while gradient descent concurrently updates $w$. The Stability Phase employs fine-grained gradient-based search with early stopping to converge to the optimal architecture. By synergizing ICSO's global navigation capability with differentiable methods' efficiency, G-ICSO-NAS achieves remarkable performance with minimal cost. In the context of the DARTS search space, an accuracy of 97.46\% is achieved on CIFAR-10 with a computational budget of just 0.15 GPU-Days. The method also exhibits strong transfer potential, recording accuracies of 83.1\% (CIFAR-100) and 75.02\% (ImageNet). Furthermore, regarding the NAS-Bench-201 benchmark, G-ICSO-NAS is shown to deliver state-of-the-art results across all evaluated datasets.

Problem

• Neural Architecture Search
• Evolutionary Algorithm
• Gradient-based NAS
• Premature Convergence
• Computational Cost

Innovation

• Neural Architecture Search
• Hybrid Optimization
• Improved Competitive Swarm Optimizer
• Differentiable Architecture Search
• Efficiency-Robustness Trade-off