InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search

2026-06-17

AI Summary

This work addresses the limitations of existing zero-cost neural architecture search methods, which rely on fragmented heuristic proxies that fail to accurately capture the intrinsic trainability of architectures. The authors propose a theoretical framework termed Intrinsic Trainability (InTrain), which formalizes trainability as an architecture-invariant property arising from the coupling of geometric capacity and optimization resilience. To model their non-additive synergy, a scale-invariant multiplicative fusion mechanism is introduced. Geometric capacity is quantified via the participation ratio of activation covariance spectra, while optimization resilience is assessed through cumulative gradient health. Leveraging insights from information geometry and backpropagation dynamics, the method enables efficient, training-free evaluation. On standard NAS benchmarks, InTrain achieves ranking correlation comparable to state-of-the-art ensemble proxies and significantly outperforms existing single-metric zero-cost approaches.

Abstract

Training-free neural architecture search promises efficient discovery of high-performance networks without costly training. However, existing zero-cost proxies rely on fragmented heuristics that fail to capture the fundamental question: what makes an architecture trainable? This paper introduces Intrinsic Trainability (InTrain), a unified theoretical proxy that formalizes trainability as an architectural invariant emerging from two synergistic components: geometric capacity and optimization resilience. We operationalize intrinsic trainability through analysis of neural information processing. Geometric capacity is quantified via the participation ratio of activation covariance eigenspectrum, capturing the effective dimensionality of representation manifolds. Optimization resilience is measured through cumulative gradient health, assessing the robustness of backpropagation across network depth. InTrain synthesizes these dimensions through a scale-invariant multiplicative coupling, which we hypothesize is essential for capturing their synergistic, non-additive relationship. Extensive experiments on standard NAS benchmarks and search spaces demonstrate that InTrain achieves ranking correlations on par with state-of-the-art ensemble-based proxies and outperforms other single-metric methods.

Problem

• neural architecture search
• trainability
• zero-cost proxy
• geometric capacity
• optimization resilience

Innovation

• Intrinsic Trainability
• Zero-Cost Neural Architecture Search
• Geometric Capacity
• Optimization Resilience
• Activation Covariance