Near Optimal Decision Trees in a SPLIT Second

2025-02-21

AI Summary

Decision trees face a fundamental trade-off between accuracy and scalability in global optimization: greedy algorithms are efficient but suboptimal, while exact methods achieve optimality at prohibitive computational cost. This paper introduces the SPLIT family of algorithms, which pioneers a depth-adaptive optimization strategy—employing greedy splitting in shallow levels for efficiency and switching to sparse look-ahead dynamic programming with heuristic pruning in deeper levels to construct near-optimal trees efficiently. SPLIT is the first method to support leaf-side approximate inference and scalable Rashomon set generation (i.e., collections of near-optimal trees). On standard benchmarks, SPLIT achieves over 99% of the test accuracy and over 80% of the model sparsity attained by globally optimal methods, while accelerating inference by several orders of magnitude; its negligible accuracy loss enables, for the first time, real-time, globally near-optimal decision tree training on large-scale datasets.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Decision tree optimization is fundamental to interpretable machine learning. The most popular approach is to greedily search for the best feature at every decision point, which is fast but provably suboptimal. Recent approaches find the global optimum using branch and bound with dynamic programming, showing substantial improvements in accuracy and sparsity at great cost to scalability. An ideal solution would have the accuracy of an optimal method and the scalability of a greedy method. We introduce a family of algorithms called SPLIT (SParse Lookahead for Interpretable Trees) that moves us significantly forward in achieving this ideal balance. We demonstrate that not all sub-problems need to be solved to optimality to find high quality trees; greediness suffices near the leaves. Since each depth adds an exponential number of possible trees, this change makes our algorithms orders of magnitude faster than existing optimal methods, with negligible loss in performance. We extend this algorithm to allow scalable computation of sets of near-optimal trees (i.e., the Rashomon set).

Problem

• Optimizing decision trees for interpretability
• Balancing accuracy and scalability
• Generating near-optimal trees efficiently

Innovation

• SPLIT algorithm optimizes decision trees
• Combines global and greedy search methods
• Enables scalable near-optimal tree computation