Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding

2025-10-21

AI Summary

Existing information-flow-based neural architecture encoders achieve high prediction accuracy but suffer from slow inference due to architectural complexity, limiting practical deployment. Method: We propose Flow-Guided Pretraining (FGP), a generative self-supervised pretraining framework that eliminates dedicated complex structures. Instead, FGP trains an encoder to reconstruct lightweight proxy representations of architectural information flow—thereby efficiently capturing information propagation patterns—while jointly optimizing information-flow modeling, architecture encoding, and reconstruction. Downstream tasks leverage supervised fine-tuning to enhance generalization. Contribution/Results: Experiments demonstrate that FGP improves Precision@1% by up to 106% over purely supervised baselines, achieving a superior trade-off between predictive accuracy and computational efficiency. By decoupling expressive information-flow modeling from rigid architectural design, FGP establishes a more practical and scalable paradigm for neural architecture performance prediction in neural architecture search.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The performance of a deep learning model on a specific task and dataset depends heavily on its neural architecture, motivating considerable efforts to rapidly and accurately identify architectures suited to the target task and dataset. To achieve this, researchers use machine learning models-typically neural architecture encoders-to predict the performance of a neural architecture. Many state-of-the-art encoders aim to capture information flow within a neural architecture, which reflects how information moves through the forward pass and backpropagation, via a specialized model structure. However, due to their complicated structures, these flow-based encoders are significantly slower to process neural architectures compared to simpler encoders, presenting a notable practical challenge. To address this, we propose FGP, a novel pre-training method for neural architecture encoding that trains an encoder to capture the information flow without requiring specialized model structures. FGP trains an encoder to reconstruct a flow surrogate, our proposed representation of the neural architecture's information flow. Our experiments show that FGP boosts encoder performance by up to 106% in Precision-1%, compared to the same encoder trained solely with supervised learning.

Problem

• Improving neural architecture encoder prediction accuracy
• Accelerating slow flow-based neural architecture processing
• Capturing information flow without specialized model structures

Innovation

• Pre-training method captures information flow
• Reconstructs flow surrogate without specialized structures
• Boosts encoder performance with supervised learning