🤖 AI Summary
Existing differentiable architecture search (DAS) methods are confined to coarse-grained operation-level optimization, neglecting fine-grained filter- and weight-level structural adaptivity, and often compromise search quality due to memory constraints. This paper proposes a multi-granularity differentiable architecture search framework that unifies modeling and jointly optimizes architectures across operation, filter, and weight levels. We introduce an adaptive granularity-aware pruning mechanism and a multi-stage progressive re-evaluation strategy to overcome fixed pruning-ratio limitations, mitigating premature convergence and error accumulation. Additionally, we design granularity-specific discrete function learning, dynamic regrowth, and subnet optimization modules. Extensive experiments on CIFAR-10/100 and ImageNet demonstrate state-of-the-art performance: our method achieves higher accuracy under comparable parameter counts, or reduces parameters by over 30% at equivalent accuracy.
📝 Abstract
Neural architecture search (NAS) has gained significant traction in automating the design of neural networks. To reduce the time cost, differentiable architecture search (DAS) transforms the traditional paradigm of discrete candidate sampling and evaluation into that of differentiable super-net optimization and discretization. However, existing DAS methods fail to trade off between model performance and model size. They either only conduct coarse-grained operation-level search, which results in redundant model parameters, or restrictively explore fine-grained filter-level and weight-level units with pre-defined remaining ratios, suffering from excessive pruning problem. Additionally, these methods compromise search quality to save memory during the search process. To tackle these issues, we introduce multi-granularity architecture search (MGAS), a unified framework which aims to discover both effective and efficient neural networks by comprehensively yet memory-efficiently exploring the multi-granularity search space. Specifically, we improve the existing DAS methods in two aspects. First, we balance the model unit numbers at different granularity levels with adaptive pruning. We learn discretization functions specific to each granularity level to adaptively determine the unit remaining ratio according to the evolving architecture. Second, we reduce the memory consumption without degrading the search quality using multi-stage search. We break down the super-net optimization and discretization into multiple sub-net stages, and perform progressive re-evaluation to allow for re-pruning and regrowing of previous units during subsequent stages, compensating for potential bias. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate that MGAS outperforms other state-of-the-art methods in achieving a better trade-off between model performance and model size.
