POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence

2025-05-16

AI Summary

This paper addresses key challenges in low-budget hyperparameter optimization (HPO)—namely, imbalanced resource allocation, high evaluation variance, and poor robustness—by proposing POCAII, a novel HPO algorithm. Methodologically, POCAII is the first to explicitly decouple the search and evaluation phases and introduces a budget-aware progressive resource reallocation mechanism: it prioritizes efficient candidate configuration generation early on and dynamically increases evaluation depth later. The algorithm integrates iterative intelligent scheduling, theory-guided exploration-exploitation strategies, and an adaptive budget allocation framework, fully compatible with black-box optimization. Empirical results demonstrate that, under tight budget constraints, POCAII significantly outperforms state-of-the-art methods—including SMAC, BOHB, and DEHB—in convergence speed, evaluation variance reduction, and robustness, making it particularly suitable for practical, high-cost model tuning tasks.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

In this paper we propose for the first time the hyperparameter optimization (HPO) algorithm POCAII. POCAII differs from the Hyperband and Successive Halving literature by explicitly separating the search and evaluation phases and utilizing principled approaches to exploration and exploitation principles during both phases. Such distinction results in a highly flexible scheme for managing a hyperparameter optimization budget by focusing on search (i.e., generating competing configurations) towards the start of the HPO process while increasing the evaluation effort as the HPO comes to an end. POCAII was compared to state of the art approaches SMAC, BOHB and DEHB. Our algorithm shows superior performance in low-budget hyperparameter optimization regimes. Since many practitioners do not have exhaustive resources to assign to HPO, it has wide applications to real-world problems. Moreover, the empirical evidence showed how POCAII demonstrates higher robustness and lower variance in the results. This is again very important when considering realistic scenarios with extremely expensive models to train.

Problem

• Optimizing hyperparameters with limited computational resources
• Improving robustness and reducing variance in HPO results
• Balancing search and evaluation phases in hyperparameter optimization

Innovation

• Separates search and evaluation phases explicitly
• Manages budget flexibly by focusing on search early
• Shows superior performance in low-budget HPO