Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study

2026-06-18

AI Summary

This study investigates how to select optimal loss function–optimizer pairings for robust training across structurally diverse neural networks. Leveraging the LEMUR heterogeneous architecture pool, the authors systematically evaluate 18 combinations of three loss functions—Cross-Entropy, Negative Log-Likelihood (NLL), and Normalized Gradient Loss (NGL)—with six optimizer families, including SGD and variants of Adam, across six image classification benchmarks. By fixing hyperparameters to control for confounding variables, they generate 594 model variants. Their large-scale empirical analysis reveals, for the first time in heterogeneous architectures, that loss–optimizer pairings significantly impact performance, challenging the notion of a universally effective combination: Cross-Entropy paired with Adam or AdamW demonstrates the most consistent robustness; NGL is only effective with adaptive optimizers and rivals Cross-Entropy in convolutional models; whereas Adagrad and Adadelta consistently underperform.

Abstract

The choice of loss function and optimizer is an important decision, that shapes further model training. Yet automated architecture search pipelines (AutoML) benefits significantly more from the optimal pairing selection and vice versa. This paper investigates whether a single recipe is sufficient for heterogeneous architecture pools, or whether the optimal pairing varies across structurally diverse models. We conduct a systematic empirical study of all $3 \times 6 = 18$ combinations of six optimizers (SGD+Momentum, Adam, AdamW, RMSprop, Adagrad, Adadelta), paired with three loss functions: Cross-Entropy (CEL), Negative Log-Likelihood (NLL), and the recently introduced genetically evolved NGL loss across the base models presented in LEMUR heterogeneous architecture pool on six image classification datasets (CelebA-Gender, CIFAR-10, CIFAR-100, ImageNette, MNIST, SVHN). The 18 loss-optimizer configurations are applied to each of the 33 compatible base architectures taken from the LEMUR pool, resulting in 594 variants that were generated fully automatically by a source-level injection pipeline and evaluated under fixed hyperparameters, ensuring that observed accuracy differences are attributable solely to the loss-optimizer pairing. Our results confirm that no single pairing is universally optimal. Cross-Entropy with Adam or AdamW is the most robust choice across architecture families and datasets. NGL is a competitive alternative to CEL on standard convolutional classifiers, but only when paired with adaptive optimizers; it degrades substantially with SGD or accumulation-based methods. Adagrad and Adadelta consistently underperform under fixed hyperparameters regardless of loss function, highlighting their sensitivity to learning rate tuning. These findings provide actionable guidance for loss-optimizer selection within NNGPT Framework.

Problem

• loss function
• optimizer
• AutoML
• heterogeneous architectures
• robust training

Innovation

• loss-optimizer pairing
• AutoML pipeline
• heterogeneous architecture pool
• NNGPT framework
• empirical robustness study