Multi-objective Hyperparameter Optimization in the Age of Deep Learning

2025-11-11

AI Summary

Existing hyperparameter optimization (HPO) methods struggle to effectively incorporate users’ prior preferences over multiple objectives—such as accuracy, latency, and model size—limiting optimization efficiency and practicality in deep learning. This paper proposes PriMO, the first method to systematically model and embed multi-objective prior knowledge into a Bayesian optimization framework. PriMO jointly models customizable multi-objective trade-off strategies and prior distributions, enabling flexible specification and integration of user preferences. Evaluated on eight deep learning benchmark tasks, PriMO consistently outperforms state-of-the-art HPO methods, delivering significant improvements in both multi-objective and single-objective HPO performance. It bridges a critical research gap in prior-driven multi-objective HPO and establishes a new paradigm for deployment-oriented deep learning hyperparameter optimization.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

While Deep Learning (DL) experts often have prior knowledge about which hyperparameter settings yield strong performance, only few Hyperparameter Optimization (HPO) algorithms can leverage such prior knowledge and none incorporate priors over multiple objectives. As DL practitioners often need to optimize not just one but many objectives, this is a blind spot in the algorithmic landscape of HPO. To address this shortcoming, we introduce PriMO, the first HPO algorithm that can integrate multi-objective user beliefs. We show PriMO achieves state-of-the-art performance across 8 DL benchmarks in the multi-objective and single-objective setting, clearly positioning itself as the new go-to HPO algorithm for DL practitioners.

Problem

• Incorporating multi-objective prior knowledge into hyperparameter optimization algorithms
• Addressing the absence of HPO methods leveraging expert beliefs across multiple objectives
• Optimizing multiple performance objectives simultaneously in deep learning workflows

Innovation

• Integrates multi-objective user beliefs into HPO
• Achieves state-of-the-art performance across benchmarks
• First HPO algorithm with multi-objective prior knowledge