Growth strategies for arbitrary DAG neural architectures

2025-01-22

AI Summary

To address the high computational cost and environmental impact induced by ever-larger deep learning models, this paper proposes Neural Architecture Growth (NAG), the first method enabling online, dynamic expansion of arbitrary directed acyclic graph (DAG)-structured architectures. NAG identifies representational bottlenecks via backpropagation gradient sensitivity, enabling goal-directed structural growth. It integrates a parameter-efficiency-driven co-design of growth and pruning, jointly optimizing training and inference efficiency. On multiple benchmark tasks, NAG achieves comparable accuracy with significantly reduced training time (average reduction of 32%) and model size (average parameter reduction of 41%), while improving FLOPs-to-parameter efficiency. Moreover, it supports interpretable, progressive architectural evolution—enabling transparent, stepwise adaptation of network topology during training.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Deep learning has shown impressive results obtained at the cost of training huge neural networks. However, the larger the architecture, the higher the computational, financial, and environmental costs during training and inference. We aim at reducing both training and inference durations. We focus on Neural Architecture Growth, which can increase the size of a small model when needed, directly during training using information from the backpropagation. We expand existing work and freely grow neural networks in the form of any Directed Acyclic Graph by reducing expressivity bottlenecks in the architecture. We explore strategies to reduce excessive computations and steer network growth toward more parameter-efficient architectures.

Problem

• Neural Network Efficiency
• Environmental Sustainability
• Complex Network Structures

Innovation

• Dynamic Expansion
• Backpropagation Optimization
• Efficient Computation