Improving Decision Trees through the Lens of Parameterized Local Search

2025-10-14

AI Summary

This paper studies the local search optimization problem for decision trees: minimizing classification error under a fixed number of local operations—such as adjusting split thresholds or swapping splitting features. Using parameterized complexity analysis, we establish for the first time that while the problem is NP-complete, it is fixed-parameter tractable (FPT) with respect to both the number of features $d$ and the maximum attribute domain size $D$. Based on this insight, we design an exact algorithm with time complexity $(D+1)^{2d} cdot |I|^{O(1)}$, where $|I|$ denotes input size. Both theoretical analysis and prototype implementation confirm that, when $d$ and $D$ are small, the algorithm efficiently computes optimal local improvements in polynomial time. This significantly enhances the feasibility and practicality of “controllable optimization” in interpretable decision tree modeling.

Technology Category

• Search and Optimization: Local Search
• Machine Learning: Optimization
• Constraint Satisfaction and Optimization: Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Algorithms for learning decision trees often include heuristic local-search operations such as (1) adjusting the threshold of a cut or (2) also exchanging the feature of that cut. We study minimizing the number of classification errors by performing a fixed number of a single type of these operations. Although we discover that the corresponding problems are NP-complete in general, we provide a comprehensive parameterized-complexity analysis with the aim of determining those properties of the problems that explain the hardness and those that make the problems tractable. For instance, we show that the problems remain hard for a small number $d$ of features or small domain size $D$ but the combination of both yields fixed-parameter tractability. That is, the problems are solvable in $(D + 1)^{2d} cdot |I|^{O(1)}$ time, where $|I|$ is the size of the input. We also provide a proof-of-concept implementation of this algorithm and report on empirical results.

Problem

• Minimizing classification errors in decision trees
• Analyzing parameterized complexity of local search operations
• Identifying tractable cases for feature-domain combinations

Innovation

• Parameterized local search for decision trees
• Fixed-parameter tractability with feature constraints
• Efficient algorithm combining domain and feature bounds