ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures

2025-10-06

AI Summary

Current neural architecture search (NAS) performance evaluation suffers from limited generalizability and scalability due to reliance on search-space-specific and handcrafted graph encodings. To address this, we propose ONNX-Net, a unified neural network representation framework that losslessly maps arbitrary architectures into standardized ONNX format and introduces the first natural-language-based textual encoding—capable of jointly representing heterogeneous layer types, operational parameters, and topological structures. Leveraging this representation, we construct ONNX-Bench, the first large-scale benchmark comprising over 600,000 architecture–accuracy pairs. Furthermore, we develop a zero-shot performance prediction surrogate by fine-tuning pretrained language models on ONNX-Net encodings. Experiments demonstrate that, with only minimal pretraining data, our surrogate achieves high-accuracy instantaneous evaluation across multiple heterogeneous search spaces—effectively breaking the long-standing search-space dependency bottleneck.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neural architecture search (NAS) automates the design process of high-performing architectures, but remains bottlenecked by expensive performance evaluation. Most existing studies that achieve faster evaluation are mostly tied to cell-based search spaces and graph encodings tailored to those individual search spaces, limiting their flexibility and scalability when applied to more expressive search spaces. In this work, we aim to close the gap of individual search space restrictions and search space dependent network representations. We present ONNX-Bench, a benchmark consisting of a collection of neural networks in a unified format based on ONNX files. ONNX-Bench includes all open-source NAS-bench-based neural networks, resulting in a total size of more than 600k {architecture, accuracy} pairs. This benchmark allows creating a shared neural network representation, ONNX-Net, able to represent any neural architecture using natural language descriptions acting as an input to a performance predictor. This text-based encoding can accommodate arbitrary layer types, operation parameters, and heterogeneous topologies, enabling a single surrogate to generalise across all neural architectures rather than being confined to cell-based search spaces. Experiments show strong zero-shot performance across disparate search spaces using only a small amount of pretraining samples, enabling the unprecedented ability to evaluate any neural network architecture instantly.

Problem

• Addressing neural architecture search evaluation bottlenecks
• Creating universal representations for diverse neural architectures
• Enabling cross-search space performance prediction with minimal data

Innovation

• Universal ONNX-based representation for neural architectures
• Text-based encoding accommodates arbitrary layer types
• Zero-shot performance prediction across disparate search spaces