Covariance-Driven Regression Trees: Reducing Overfitting in CART

2026-01-12

AI Summary

This work addresses the tendency of conventional CART regression trees to overfit, particularly under deep growth or small-sample conditions, which compromises their generalization performance. To mitigate this issue, the authors propose a covariance-driven splitting criterion, termed CovRT, which replaces empirical risk minimization with the covariance between predictors and the response variable as the basis for node splitting. This mechanism inherently promotes more balanced and stable tree structures while enhancing the identification of true signal variables. Theoretical analysis establishes that CovRT satisfies an oracle inequality and achieves a convergence rate comparable to CART in high-dimensional settings. Empirical evaluations on both simulated and real-world datasets demonstrate that CovRT significantly improves predictive accuracy and effectively alleviates overfitting.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Decision trees are powerful machine learning algorithms, widely used in fields such as economics and medicine for their simplicity and interpretability. However, decision trees such as CART are prone to overfitting, especially when grown deep or the sample size is small. Conventional methods to reduce overfitting include pre-pruning and post-pruning, which constrain the growth of uninformative branches. In this paper, we propose a complementary approach by introducing a covariance-driven splitting criterion for regression trees (CovRT). This method is more robust to overfitting than the empirical risk minimization criterion used in CART, as it produces more balanced and stable splits and more effectively identifies covariates with true signals. We establish an oracle inequality of CovRT and prove that its predictive accuracy is comparable to that of CART in high-dimensional settings. We find that CovRT achieves superior prediction accuracy compared to CART in both simulations and real-world tasks.

Problem

• overfitting
• decision trees
• CART
• regression trees
• covariance

Innovation

• covariance-driven splitting
• regression trees
• overfitting reduction
• CART
• oracle inequality