BTTackler: A Diagnosis-based Framework for Efficient Deep Learning Hyperparameter Optimization

2024-08-24

Knowledge Discovery and Data Mining

AI Summary

This work addresses the inefficiency in deep learning hyperparameter optimization caused by the inability to identify ineffective trials—such as those suffering from vanishing gradients or insufficient convergence—during early training stages, which leads to significant computational waste. To mitigate this, we propose BTTackler, a novel framework that integrates training diagnostics directly into the hyperparameter optimization pipeline. BTTackler employs quantitative metrics to automatically detect training anomalies and trigger early stopping, thereby conserving resources. The framework is compatible with mainstream optimizers and neural architectures and is accompanied by a lightweight open-source library. Empirical results demonstrate that BTTackler reduces average optimization time by 40.33% while achieving comparable accuracy, and under fixed time budgets, it completes 44.5% more high-quality (Top-10) trials than baseline methods.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Hyperparameter optimization (HPO) is known to be costly in deep learning, especially when leveraging automated approaches. Most of the existing automated HPO methods are accuracy-based, i.e., accuracy metrics are used to guide the trials of different hyperparameter configurations amongst a specific search space. However, many trials may encounter severe training problems, such as vanishing gradients and insufficient convergence, which can hardly be reflected by accuracy metrics in the early stages of the training and often result in poor performance. This leads to an inefficient optimization trajectory because the bad trials occupy considerable computation resources and reduce the probability of finding excellent hyperparameter configurations within a time limitation. In this paper, we propose Bad Trial Tackler (BTTackler), a novel HPO framework that introduces training diagnosis to identify training problems automatically and hence tackles bad trials. BTTackler diagnoses each trial by calculating a set of carefully designed quantified indicators and triggers early termination if any training problems are detected. Evaluations are performed on representative HPO tasks consisting of three classical deep neural networks (DNN) and four widely used HPO methods. To better quantify the effectiveness of an automated HPO method, we propose two new measurements based on accuracy and time consumption. Results show the advantage of BTTackler on two-fold: (1) it reduces 40.33% of time consumption to achieve the same accuracy comparable to baseline methods on average and (2) it conducts 44.5% more top-10 trials than baseline methods on average within a given time budget. We also released an open-source Python library that allows users to easily apply BTTackler to automated HPO processes with minimal code changes\footnotehttps://github.com/thuml/BTTackler.

Problem

• Hyperparameter Optimization
• Training Diagnosis
• Deep Learning
• Early Termination
• Efficiency

Innovation

• hyperparameter optimization
• training diagnosis
• early termination
• deep learning efficiency
• automated machine learning