Gradient-Descent Steps to Success over Mean Accuracy: A Paradigm Shift for ML

2026-06-20

AI Summary

This work addresses the conventional overemphasis on final accuracy in machine learning evaluation while neglecting the computational cost required to achieve it. The authors propose a new efficiency-oriented AutoML paradigm centered on the number of gradient descent steps as a core metric, quantifying the expected computational effort needed for a model to reach a target accuracy with high probability. Extending this perspective beyond genetic programming to all gradient-based models, they theoretically and empirically demonstrate—across 11 models and 5 datasets—that large learning rates can substantially reduce expected computational overhead. Furthermore, they show that the optimal training strategy shifts between a single long run and multiple short restarts depending on the accuracy target, leading to a practical framework for model selection under fixed computational budgets.

Abstract

Traditional evaluation of machine learning (ML) models typically focuses on achieving the maximum possible accuracy irrespective of the computational cost. In this article, we propose a paradigm shift towards evaluating performance based on computational effort-explicitly defined here as the total number of gradient descent steps required to reach an acceptable level of accuracy with high probability. Building upon the concept of computational effort originally introduced by Koza for Genetic Programming, we extend this metric to any ML model trained via gradient descent. Furthermore, we demonstrate that minimising this effort acts as a novel form of Automatic Machine Learning (AutoML). By evaluating it across 11 diverse ML models and five standard classification datasets, we uncover significant insights into the dynamics of gradient-based learning. Our findings reveal that optimal hyper-parameters consistently favour unusually large learning rates. Crucially, we demonstrate that the rapid, aggressive landscape traversal enabled by these large rates not only promotes generalisation-as seen in phenomena like superconvergence-but also statistically minimises the expected computational effort for training. Furthermore, we identify distinct phase transitions in the optimal search strategy: while a single training run suffices for lower accuracy targets, reaching a model's performance limit requires a dramatic shift towards conducting numerous independent, short restarts. Finally, we illustrate how this effort-based paradigm provides a robust framework for model selection, allowing practitioners to choose optimal algorithms based on the difficulty of a problem as perceived by different models for a given target accuracy, or to maximise the achievable accuracy for a fixed budget of gradient descent steps.

Problem

• computational effort
• gradient descent
• model evaluation
• hyper-parameter optimization
• AutoML

Innovation

• computational effort
• gradient descent steps
• large learning rates
• phase transitions
• effort-based model selection