🤖 AI Summary
This work addresses the challenge of optimizing the number of trees in random forests, a hyperparameter whose performance typically improves monotonically with ensemble size, thereby confounding conventional hyperparameter optimization methods due to sensitivity to preset bounds or early-stopping noise. The authors propose a triplet plateau search algorithm that excludes this parameter from the Tree-structured Parzen Estimator (TPE) search space and instead employs a sliding window to monitor relative changes in out-of-bag (OOB) scores across three consecutive ensemble sizes. This approach adaptively identifies the minimal ensemble size satisfying a target performance criterion, eliminating the need for a predefined upper bound while offering both automation and interpretability. Theoretical analysis links the plateau criterion to the asymptotic performance gap and variance estimation. Empirical results demonstrate substantial reductions in tree count on most datasets, while automatically selecting larger ensembles on high-dimensional biological benchmarks such as Arcene and Dorothea, confirming the method’s adaptivity and effectiveness.
📝 Abstract
Hyperparameter optimization (HPO) for Random Forest faces a specific difficulty in tuning the number of trees: the predictive score typically improves monotonically with ensemble size, so standard methods such as Tree-structured Parzen Estimator (TPE) and Hyperband require a predefined search range and often drive the estimate toward its right boundary. Early-stopping strategies avoid fixing such a range, but can be sensitive to score noise and prone to premature stopping. To address this, we propose an integrated triplet-based plateau-search algorithm that removes the number of trees from the direct TPE search space and still exploits information accumulated across HPO trials. The method adaptively tracks a near-minimal sufficient ensemble size by monitoring relative changes in the out-of-bag (OOB) score across a triplet of forest sizes and shifting this triplet accordingly. This yields an automated and user-interpretable procedure based on a tolerance parameter. We also provide a theoretical analysis: we relate the proposed relative OOB-score criterion to the gap between the current and limiting scores, and derive an asymptotic variance estimate for the corresponding OOB-based absolute relative difference. Experiments show that the selected number of trees can differ substantially from the common heuristic: for most classical benchmark datasets it is smaller, whereas for some high-dimensional bioinformatics datasets, such as Arcene and Dorothea, it is larger. The source code and reproducible experiments are available at https://github.com/lange-am/rf_plateau_hpo.