How far away are truly hyperparameter-free learning algorithms?

2025-05-29

AI Summary

Deep learning training typically requires costly, task-specific hyperparameter tuning—especially of the learning rate—hindering scalable and automated model development. Method: This work systematically evaluates learning-rate-free optimizers as universal “zero-configuration” training algorithms across diverse tasks, adopting a workload-agnostic paradigm. We conduct unified calibration and cross-task generalization evaluation using the AlgoPerf benchmark, with rigorous comparison against a jointly tuned NadamW baseline under frozen default configurations and normalized performance scoring. Contribution/Results: Calibrated learning-rate-free methods achieve substantial performance gains over uncalibrated variants; however, they still fall short of the jointly optimized NadamW baseline across the full task spectrum. This demonstrates that while learning-rate-free optimization is promising and practically improved through systematic calibration, fully hyperparameter-free neural network training remains an open research challenge.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Despite major advances in methodology, hyperparameter tuning remains a crucial (and expensive) part of the development of machine learning systems. Even ignoring architectural choices, deep neural networks have a large number of optimization and regularization hyperparameters that need to be tuned carefully per workload in order to obtain the best results. In a perfect world, training algorithms would not require workload-specific hyperparameter tuning, but would instead have default settings that performed well across many workloads. Recently, there has been a growing literature on optimization methods which attempt to reduce the number of hyperparameters -- particularly the learning rate and its accompanying schedule. Given these developments, how far away is the dream of neural network training algorithms that completely obviate the need for painful tuning? In this paper, we evaluate the potential of learning-rate-free methods as components of hyperparameter-free methods. We freeze their (non-learning rate) hyperparameters to default values, and score their performance using the recently-proposed AlgoPerf: Training Algorithms benchmark. We found that literature-supplied default settings performed poorly on the benchmark, so we performed a search for hyperparameter configurations that performed well across all workloads simultaneously. The best AlgoPerf-calibrated learning-rate-free methods had much improved performance but still lagged slightly behind a similarly calibrated NadamW baseline in overall benchmark score. Our results suggest that there is still much room for improvement for learning-rate-free methods, and that testing against a strong, workload-agnostic baseline is important to improve hyperparameter reduction techniques.

Problem

• Eliminating hyperparameter tuning in machine learning algorithms
• Assessing performance of learning-rate-free optimization methods
• Improving workload-agnostic defaults for neural network training

Innovation

• Evaluates learning-rate-free optimization methods
• Uses AlgoPerf benchmark for performance scoring
• Searches for workload-agnostic hyperparameter configurations