model architecture & optimization

This work addresses the limitations of existing neural architecture search (NAS) methods, which are either confined to narrow predefined search spaces or suffer from inefficiency and bias when leveraging large language models (LLMs) for open-ended exploration. To overcome these challenges, the authors propose a semi-automated NAS framework that constructs a prior-informed, open search space by structurally modeling architectural knowledge extracted from scientific literature. The framework integrates the FairNAD algorithm with multiple fairness-aware mutation mechanisms—including fair sampling, Pareto-aware selection, and LLM-driven iterative refinement—to enable efficient, diverse, and high-quality architecture discovery. Empirical evaluations demonstrate consistent improvements over state-of-the-art methods, achieving accuracy gains of 0.84%, 2.17%, and 2.35% on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively.

One-shot neural architecture search (NAS) methods, such as DARTS, suffer from high GPU memory consumption and low search efficiency due to the large candidate search space. To address this, this paper proposes a zero-shot NAS-based automatic pre-pruning strategy for the search space: it first employs zero-shot proxies—computationally efficient, training-free metrics—to identify and eliminate low-performing sub-architectures; then performs differentiable one-shot search over the pruned space. This is the first work to leverage zero-shot NAS for search space pre-pruning. On the DARTS benchmark, our method reduces GPU memory usage by 81%, significantly accelerates the search process, and shrinks the search space by 50%, while achieving final architecture accuracy on par with that obtained from full-space search. The approach thus achieves a balanced optimization across search efficiency, memory footprint, and model performance.

To address excessive computational overhead when deploying vision models on resource-constrained devices, this paper proposes an efficient Vision Transformer (ViT) architecture design framework. Methodologically: (1) it optimizes the input-output data pathway to enhance representational capacity of lightweight models; (2) it restructures the context window of computationally constrained attention mechanisms to improve local-global modeling efficiency; and (3) it leverages the invertibility and explicit probabilistic modeling properties of normalizing flows to enable high-fidelity, low-overhead knowledge distillation. Experiments demonstrate that the proposed approach achieves comparable or superior accuracy on benchmarks such as ImageNet, while requiring significantly fewer parameters and FLOPs. It also substantially reduces inference latency and memory footprint. The framework establishes a scalable new paradigm for efficient visual understanding at the edge.

Neural architecture design often relies on heuristic rules or expensive search strategies, lacking a principled, differentiable mapping from performance to structure. Method: This paper proposes an automatic architecture optimization framework grounded in structure–performance mapping modeling. We introduce the Architecture Synthesis Neural Network (ASNN), the first model that takes a performance distribution (e.g., accuracy distribution) as input and differentiably synthesizes high-performing architectural parameters—enabling invertible, generalizable mapping from performance to structure. Leveraging a TensorFlow-based multi-layer network performance dataset, ASNN learns this inverse mapping via neural regression and incorporates an iterative prediction mechanism for progressive refinement. Contribution/Results: On both two- and three-layer networks, ASNN discovers novel architectures surpassing the original dataset’s best-performing models, achieving statistically significant average test accuracy improvements. Experiments validate its effectiveness in architecture recommendation, cross-architecture generalization, and iterative optimization.

To address the prohibitively high computational cost of training candidate architectures in neural architecture search (NAS), this paper proposes NASGraph—a zero-cost, data-agnostic lightweight method. Its core innovation lies in modeling neural networks as graphs and, for the first time, employing a graph topological feature—average degree—as a performance proxy, thereby eliminating both training and data dependencies. On NAS-Bench-201, NASGraph identifies the optimal architecture among 200 randomly sampled candidates in just 217 seconds with high accuracy. It further achieves competitive ranking correlation across multiple benchmarks, including NAS-Bench-101, NDS, and Micro TransNAS-Bench-101. By decoupling architecture evaluation from training and data, NASGraph significantly improves evaluation efficiency and cross-benchmark transferability, offering a scalable and practical solution for resource-constrained NAS.

Current neural architecture search (NAS) performance evaluation suffers from limited generalizability and scalability due to reliance on search-space-specific and handcrafted graph encodings. To address this, we propose ONNX-Net, a unified neural network representation framework that losslessly maps arbitrary architectures into standardized ONNX format and introduces the first natural-language-based textual encoding—capable of jointly representing heterogeneous layer types, operational parameters, and topological structures. Leveraging this representation, we construct ONNX-Bench, the first large-scale benchmark comprising over 600,000 architecture–accuracy pairs. Furthermore, we develop a zero-shot performance prediction surrogate by fine-tuning pretrained language models on ONNX-Net encodings. Experiments demonstrate that, with only minimal pretraining data, our surrogate achieves high-accuracy instantaneous evaluation across multiple heterogeneous search spaces—effectively breaking the long-standing search-space dependency bottleneck.

This work addresses the computational, memory, and storage bottlenecks associated with deploying large-scale deep neural networks (DNNs) in resource-constrained environments. The authors propose a novel pruning method that integrates system-level engineering requirements with human-interpretable concepts—such as color and semantic categories—to identify critical neurons through analysis of their activation patterns, thereby guiding the generation of lightweight models. Notably, this approach is the first to incorporate interpretable concepts directly into the DNN pruning pipeline. Evaluated on VGG-19 using a dataset comprising 26,384 RGB images, the method yields pruned models that achieve substantial reductions in model size and computational overhead while maintaining high performance, demonstrating strong applicability across diverse real-world scenarios with stringent resource constraints.

Neural Architecture Search (NAS) has emerged as a powerful framework for automatically discovering neural architectures that balance accuracy and efficiency. However, as AI transitions from static benchmarks to real-world deployment, the traditional focus on hardware-aware efficiency is no longer sufficient. We observe that modern NAS methods, especially those that target edge AI, are evolving to address a triple objective: Efficiency, Robustness, and Continual Learning. While efficiency ensures feasibility in resource-constrained environments, robustness guarantees reliability under environmental variabilities, and continual learning enables adaptation to sequential tasks without catastrophic forgetting. We propose a taxonomy of NAS approaches through this triple lens, distinguishing between methods targeting resource optimization, environmental resilience, and architectural plasticity. This unified perspective reveals that these axes, though often studied in isolation, are mutually reinforcing. Building on this taxonomy, we map the current landscape of these NAS methods into a new framework called Hardware-Efficient, Robust, and ContinUal LEarning Search (HERCULES). We define the desiderata, the twelve labours of HERCULES, addressing the non-trivial challenge of balancing an adequate search-space exploration with the immense computational costs of a multi-objective NAS, accounting for these crucial objectives of current AI systems. By identifying critical gaps in existing research, this survey outlines a roadmap toward integrated algorithmic, architectural, and hardware-software co-design for truly deployable, lifelong-learning AI systems.

Hardware-aware neural architecture search (HW-NAS) faces dual challenges: joint optimization of accuracy and latency, and low search efficiency. Conventional supernet-based methods incur prohibitive computational overhead, while existing LLM-driven approaches suffer from exploration bias, limiting coverage of diverse architectures across latency regimes. This paper proposes an efficient LLM-driven HW-NAS framework. Its core innovations are: (1) a complexity-aware hierarchical partitioning of the search space; and (2) a co-evolutionary mechanism for architecture generation and prompt engineering, integrating zero-cost predictors and knowledge-base-guided prompt refinement to mitigate LLMs’ intrinsic architectural biases. Evaluated on HW-NAS-Bench, our method achieves higher hypervolume (HV), lower inverted generational distance (IGD), up to 54% latency reduction, and compresses search time from days to minutes—significantly improving both efficiency and diversity in discovering cost-effective architectures across latency ranges.