Loss Functions for Predictor-based Neural Architecture Search

2025-06-06

AI Summary

The design of loss functions for neural architecture search (NAS) performance predictors lacks systematic investigation. This work presents the first taxonomy and unified empirical evaluation of eight loss functions—spanning regression (e.g., MSE), ranking (pairwise and listwise), and weighted variants—across 13 tasks in five mainstream search spaces, ensuring full reproducibility. Within a predictor-based NAS framework, we demonstrate that multi-loss collaboration—particularly hybrid strategies—significantly improves both prediction accuracy and search efficiency, yielding an average 12.7% gain in Top-1 architecture discovery rate. Our study bridges critical theoretical and practical gaps in NAS predictor loss design, offering a task-adaptive, reproducible guideline for loss function selection.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Multiagent Systems: Agent/AI Theories and Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Evaluation is a critical but costly procedure in neural architecture search (NAS). Performance predictors have been widely adopted to reduce evaluation costs by directly estimating architecture performance. The effectiveness of predictors is heavily influenced by the choice of loss functions. While traditional predictors employ regression loss functions to evaluate the absolute accuracy of architectures, recent approaches have explored various ranking-based loss functions, such as pairwise and listwise ranking losses, to focus on the ranking of architecture performance. Despite their success in NAS, the effectiveness and characteristics of these loss functions have not been thoroughly investigated. In this paper, we conduct the first comprehensive study on loss functions in performance predictors, categorizing them into three main types: regression, ranking, and weighted loss functions. Specifically, we assess eight loss functions using a range of NAS-relevant metrics on 13 tasks across five search spaces. Our results reveal that specific categories of loss functions can be effectively combined to enhance predictor-based NAS. Furthermore, our findings could provide practical guidance for selecting appropriate loss functions for various tasks. We hope this work provides meaningful insights to guide the development of loss functions for predictor-based methods in the NAS community.

Problem

• Evaluating effectiveness of loss functions in NAS predictors
• Comparing regression, ranking, and weighted loss functions
• Guiding loss function selection for predictor-based NAS

Innovation

• Comprehensive study on predictor loss functions
• Combining regression and ranking loss functions
• Practical guidance for loss function selection