Simplifying Random Forests' Probabilistic Forecasts

2024-08-22

AI Summary

To address the poor interpretability of probabilistic predictions from Random Forests (RFs), this paper proposes a lightweight, retraining-free sparsification method. Leveraging a nearest-neighbor interpretation of RFs, the approach retains only the *k* most influential neighbor samples—i.e., those with the largest weights—for each prediction, setting all others to zero. This is the first systematic integration of sparse nearest-neighbor principles into RF probability calibration. Theoretically, we prove that the method achieves statistical advantages when the underlying data distribution exhibits numerous small-weight instances. Empirically, the sparsified model matches or surpasses the original RF’s predictive performance across multiple benchmark tasks, while substantially enhancing human interpretability, decision transparency, and deployment efficiency. The core contribution lies in a novel post-hoc framework that simultaneously preserves generalization capability and delivers intuitive, sparse explanations—without architectural modification or retraining.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Since their introduction by Breiman, Random Forests (RFs) have proven to be useful for both classification and regression tasks. The RF prediction of a previously unseen observation can be represented as a weighted sum of all training sample observations. This nearest-neighbor-type representation is useful, among other things, for constructing forecast distributions (Meinshausen, 2006). In this paper, we consider simplifying RF-based forecast distributions by sparsifying them. That is, we focus on a small subset of nearest neighbors while setting the remaining weights to zero. This sparsification step greatly improves the interpretability of RF predictions. It can be applied to any forecasting task without re-training existing RF models. In empirical experiments, we document that the simplified predictions can be similar to or exceed the original ones in terms of forecasting performance. We explore the statistical sources of this finding via a stylized analytical model of RFs. The model suggests that simplification is particularly promising if the unknown true forecast distribution contains many small weights that are estimated imprecisely.

Problem

• Sparsifying random forests' probabilistic forecast distributions
• Improving interpretability by focusing on top k neighbors
• Maintaining or enhancing forecasting performance through simplification

Innovation

• Sparsify RF forecast distributions via Top-k
• Improve interpretability without retraining models
• Use nearest neighbors subset for simplification