🤖 AI Summary
Existing multi-objective hyperparameter optimization (MO-HPO) methods evaluate trade-offs only at final training epochs, neglecting dynamic performance compromises that arise earlier—e.g., at overfitting onset. This work is the first to explicitly model training epoch as an optimization variable—not merely a fixed auxiliary parameter. We propose a trajectory-aware multi-objective Bayesian optimization (TA-MOBO) framework that jointly models predictive performance sequences and integrates adaptive early stopping, enabling precise identification and efficient exploitation of Pareto-optimal trade-offs during early-to-mid training. Our approach introduces a novel trajectory-aware acquisition function and a multi-objective early-stopping strategy. Evaluated on synthetic benchmarks and standard hyperparameter tuning tasks, TA-MOBO significantly improves Pareto front quality while reducing average training cost by 37%, outperforming state-of-the-art MO-HPO methods.
📝 Abstract
Training machine learning models inherently involves a resource-intensive and noisy iterative learning procedure that allows epoch-wise monitoring of the model performance. However, the insights gained from the iterative learning procedure typically remain underutilized in multi-objective hyperparameter optimization scenarios. Despite the limited research in this area, existing methods commonly identify the trade-offs only at the end of model training, overlooking the fact that trade-offs can emerge at earlier epochs in cases such as overfitting. To bridge this gap, we propose an enhanced multi-objective hyperparameter optimization problem that treats the number of training epochs as a decision variable, rather than merely an auxiliary parameter, to account for trade-offs at an earlier training stage. To solve this problem and accommodate its iterative learning, we then present a trajectory-based multi-objective Bayesian optimization algorithm characterized by two features: 1) a novel acquisition function that captures the improvement along the predictive trajectory of model performances over epochs for any hyperparameter setting and 2) a multi-objective early stopping mechanism that determines when to terminate the training to maximize epoch efficiency. Experiments on synthetic simulations and hyperparameter tuning benchmarks demonstrate that our algorithm can effectively identify the desirable trade-offs while improving tuning efficiency.