Grouped Sequential Optimization Strategy -- the Application of Hyperparameter Importance Assessment in Deep Learning

2025-03-07

AI Summary

Hyperparameter optimization (HPO) for deep learning suffers from high computational cost and inefficient search, particularly in high-dimensional spaces where Bayesian optimization remains limited. To address this, we propose a Group-Sequential HPO framework grounded in Hyperparameter Importance Assessment (HIA), introducing the novel “sequential grouping” paradigm. This approach dynamically ranks hyperparameters using prior importance weights and optimizes them in stages—partitioning the hyperparameter space into subsets and coupling HIA, Bayesian optimization, and serialized search. The method significantly improves search efficiency without compromising model performance. Experiments across six new image classification benchmarks demonstrate an average 31.9% reduction in optimization time while preserving classification accuracy. Our work establishes a new paradigm for efficient, interpretable, and resource-conscious deep learning hyperparameter tuning.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Hyperparameter optimization (HPO) is a critical component of machine learning pipelines, significantly affecting model robustness, stability, and generalization. However, HPO is often a time-consuming and computationally intensive task. Traditional HPO methods, such as grid search and random search, often suffer from inefficiency. Bayesian optimization, while more efficient, still struggles with high-dimensional search spaces. In this paper, we contribute to the field by exploring how insights gained from hyperparameter importance assessment (HIA) can be leveraged to accelerate HPO, reducing both time and computational resources. Building on prior work that quantified hyperparameter importance by evaluating 10 hyperparameters on CNNs using 10 common image classification datasets, we implement a novel HPO strategy called 'Sequential Grouping.' That prior work assessed the importance weights of the investigated hyperparameters based on their influence on model performance, providing valuable insights that we leverage to optimize our HPO process. Our experiments, validated across six additional image classification datasets, demonstrate that incorporating hyperparameter importance assessment (HIA) can significantly accelerate HPO without compromising model performance, reducing optimization time by an average of 31.9% compared to the conventional simultaneous strategy.

Problem

• Accelerate hyperparameter optimization using importance assessment.
• Reduce computational resources in deep learning model training.
• Improve efficiency of HPO in high-dimensional search spaces.

Innovation

• Sequential Grouping strategy for HPO acceleration
• Hyperparameter Importance Assessment reduces optimization time
• Leverages importance weights to enhance model performance