🤖 AI Summary
Random forests achieve high predictive accuracy but suffer from poor interpretability. Method: This paper proposes a pruning framework that significantly enhances interpretability while preserving predictive performance. We establish, for the first time, the asymptotic statistical advantage of Lasso-pruned forests and derive finite-sample generalization bounds for mainstream pruning methods. Notably, we identify cases where pruned ensembles can be exactly represented by a single decision tree—achieving a qualitative leap in interpretability. Results: Extensive validation via Monte Carlo simulations and 19 benchmark datasets (16 synthetic, 3 real) shows that pruned forests attain or surpass the accuracy of full forests using only 5–20% of the original trees; in several scenarios, compression into a single tree retains high accuracy. Our core contributions are: (i) theoretical foundations for pruned forests, (ii) rigorous finite-sample generalization analysis, and (iii) principled co-optimization of interpretability and predictive accuracy.
📝 Abstract
Regression forests have long delivered state-of-the-art accuracy, often outperforming regression trees and even neural networks, but they suffer from limited interpretability as ensemble methods. In this work, we revisit forest pruning, an approach that aims to have the best of both worlds: the accuracy of regression forests and the interpretability of regression trees. This pursuit, whose foundation lies at the core of random forest theory, has seen vast success in empirical studies. In this paper, we contribute theoretical results that support and qualify those empirical findings; namely, we prove the asymptotic advantage of a Lasso-pruned forest over its unpruned counterpart under weak assumptions, as well as high-probability finite-sample generalization bounds for regression forests pruned according to the main methods, which we then validate by way of simulation. Then, we test the accuracy of pruned regression forests against their unpruned counterparts on 19 different datasets (16 synthetic, 3 real). We find that in the vast majority of scenarios tested, there is at least one forest-pruning method that yields equal or better accuracy than the original full forest (in expectation), while just using a small fraction of the trees. We show that, in some cases, the reduction in the size of the forest is so dramatic that the resulting sub-forest can be meaningfully merged into a single tree, obtaining a level of interpretability that is qualitatively superior to that of the original regression forest, which remains a black box.