OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search

2025-09-06

AI Summary

Neural architecture search (NAS) faces challenges stemming from the discrete, vast, non-convex, and computationally expensive nature of the architecture search space. To address this, we propose an end-to-end differentiable NAS framework centered on an **Optimization Proxy** mechanism: it maps the discrete architecture space onto a continuous, differentiable, and smooth parametric manifold, enabling direct optimization of architectures via arbitrary gradient-based optimizers. Crucially, our approach eliminates reliance on surrogate predictors or hypernetworks, thereby avoiding both surrogate modeling error and bias introduced by differentiable relaxations. We evaluate the method across 12 diverse tasks spanning computer vision, natural language processing, and resource-constrained settings. Results demonstrate substantial improvements in search efficiency—3× to 10× speedup—while maintaining strong robustness and generalization even under low-fidelity evaluation. The framework is both broadly applicable and practically deployable.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Neural architecture search (NAS) is a hard computationally expensive optimization problem with a discrete, vast, and spiky search space. One of the key research efforts dedicated to this space focuses on accelerating NAS via certain proxy evaluations of neural architectures. Different from the prevalent predictor-based methods using surrogate models and differentiable architecture search via supernetworks, we propose an optimization proxy to streamline the NAS as an end-to-end optimization framework, named OptiProxy-NAS. In particular, using a proxy representation, the NAS space is reformulated to be continuous, differentiable, and smooth. Thereby, any differentiable optimization method can be applied to the gradient-based search of the relaxed architecture parameters. Our comprehensive experiments on $12$ NAS tasks of $4$ search spaces across three different domains including computer vision, natural language processing, and resource-constrained NAS fully demonstrate the superior search results and efficiency. Further experiments on low-fidelity scenarios verify the flexibility.

Problem

• Accelerates neural architecture search via optimization proxy
• Reformulates NAS space as continuous and differentiable
• Enables gradient-based search for architecture parameters

Innovation

• Optimization proxy for end-to-end NAS framework
• Reformulates search space as continuous and differentiable
• Enables gradient-based search on relaxed architecture parameters