Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

2026-07-09

AI Summary

Learning rate scheduling strategies significantly influence neural network performance, yet their selection typically relies on extensive manual trial and error. This work presents the first large-scale, systematic quantification of scheduler effectiveness across heterogeneous architectures. Leveraging the LEMUR dataset, we evaluate 25 configurations from nine PyTorch learning rate schedulers on 30 diverse convolutional and Transformer architectures, training a total of 3,938 model variants through automated source-code instrumentation. Our results reveal strong dependencies between scheduler choice and model architecture, with CosineAnnealingWarmRestarts and CyclicLR substantially outperforming conventional decay strategies. The best-performing configuration achieves a Top-1 accuracy of 86.45%, and 237 variants exceed 80% accuracy. All results are publicly released to support community research.

Abstract

Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures. We evaluated 30 representative architectures from convolutional and transformer families within the LEMUR neural network dataset. Through automated source-code injection, we applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10. Our best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%. The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies. The resulting accuracy landscape, contributed to the LEMUR nn-dataset, provides a practical reference for principled scheduler selection.

Problem

• learning rate scheduling
• neural network architectures
• classification accuracy
• heterogeneous architectures
• scheduler selection

Innovation

• learning rate scheduling
• heterogeneous architectures
• systematic evaluation
• automated source-code injection
• LEMUR dataset