Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests

2024-02-20

arXiv.org

AI Summary

This paper investigates the mechanism by which Random Forests (RF) reduce prediction bias under low signal-to-noise ratio (SNR) conditions and identifies the source of RF’s superiority over Bagging. Through systematic control of SNR, bias–variance decomposition, and comparative analysis of generalization error between RF and Bagging, we find that RF’s feature subsampling (governed by the *mtry* parameter) serves not only as a variance-reducing regularizer but also enhances detection of weak signal patterns—thereby actively reducing bias. This dual effect enables synergistic bias–variance optimization, particularly pronounced at medium-to-high SNR levels. Crucially, we provide the first empirical evidence that *mtry* tuning is a pivotal lever for balancing the bias–variance trade-off. These findings uncover the intrinsic mechanism underlying RF’s capacity for both strong model fitting and robust generalization.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Graphical Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We study the often overlooked phenomenon, first noted in cite{breiman2001random}, that random forests appear to reduce bias compared to bagging. Motivated by an interesting paper by cite{mentch2020randomization}, where the authors explain the success of random forests in low signal-to-noise ratio (SNR) settings through regularization, we explore how random forests can capture patterns in the data that bagging ensembles fail to capture. We empirically demonstrate that in the presence of such patterns, random forests reduce bias along with variance and can increasingly outperform bagging ensembles when SNR is high. Our observations offer insights into the real-world success of random forests across a range of SNRs and enhance our understanding of the difference between random forests and bagging ensembles. Our investigations also yield practical insights into the importance of tuning $mtry$ in random forests.

Problem

• Random forests reduce bias and variance compared to bagging
• Random forests outperform bagging in high signal-to-noise settings
• Tuning mtry is crucial for random forests' performance

Innovation

• Random forests reduce bias and variance
• Random forests outperform bagging in high SNR
• Tuning mtry is crucial in random forests