Optimal or Greedy Decision Trees? Revisiting their Objectives, Tuning, and Performance

2024-09-19

arXiv.org

AI Summary

This paper systematically investigates the performance gap between optimal decision trees (ODTs) and greedy decision trees, and its underlying causes. Addressing three core challenges—objective function design, hyperparameter optimization, and generalization—we conduct a large-scale empirical study across 180 benchmark datasets. First, we identify objective function concavity as a critical factor governing relative performance—previously unreported. Second, we propose a dedicated multi-objective hyperparameter tuning paradigm for ODTs, incorporating 11 distinct objectives and 6 optimization strategies. Third, leveraging combinatorial optimization, interpretability analysis, and rigorous statistical significance testing, we resolve several longstanding controversies in the literature: under proper tuning, ODTs consistently outperform greedy methods—especially under non-concave objectives, where greedy approaches frequently fail. All code and benchmarking infrastructure are open-sourced to enable reproducible, standardized evaluation.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Heuristic Search
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Recently there has been a surge of interest in optimal decision tree (ODT) methods that globally optimize accuracy directly, in contrast to traditional approaches that locally optimize an impurity or information metric. However, the value of optimal methods is not well understood yet, as the literature provides conflicting results, with some demonstrating superior out-of-sample performance of ODTs over greedy approaches, while others show the opposite. Through a novel extensive experimental study, we provide new insights into the design and behavior of learning decision trees. In particular, we identify and analyze two relatively unexplored aspects of ODTs: the objective function used in training trees, and tuning techniques. Thus, we address these three questions: what objective to optimize in ODTs; how to tune ODTs; and how do optimal and greedy methods compare? Our experimental evaluation examines 11 objective functions, six tuning methods, and six claims from the literature on optimal and greedy methods on 180 real and synthetic data sets. Through our analysis, both conceptually and experimentally, we show the effect of (non-)concave objectives in greedy and optimal approaches; we highlight the importance of proper tuning of ODTs; support and refute several claims from the literature; provide clear recommendations for researchers and practitioners on the usage of greedy and optimal methods; and code for future comparisons.

Problem

• Compare performance of optimal vs greedy decision trees
• Evaluate objective functions for optimal decision trees
• Assess tuning techniques for optimal decision trees

Innovation

• Globally optimizes accuracy directly
• Examines 11 objective functions
• Highlights importance of proper tuning