Learning Small Decision Trees with Few Outliers: A Parameterized Perspective

2024-03-24

AAAI Conference on Artificial Intelligence

AI Summary

This paper studies learning small decision trees tolerant to at most $t$ misclassifications, focusing on two core variants: minimizing tree size (DTSO) and minimizing tree depth (DTDO). Within the parameterized complexity framework, we establish for the first time that both problems are W[1]-hard with respect to $s+y$ and $d+y$, where $y$ is the number of attributes. However, when parameterized by the misclassification tolerance $t$, both become fixed-parameter tractable (FPT), and we present the first FPT algorithm explicitly depending on $t$. We systematically characterize kernelization complexity, providing tight polynomial kernel existence and impossibility results, thereby completing the kernelization classification for DTSO and DTDO. Our main contributions are: (i) establishing precise computational complexity boundaries; (ii) revealing the “complexity-reducing” role of $t$, which shifts hardness from W[1]-hardness to FPT; and (iii) delivering theoretically complete algorithms and matching lower bounds.

Technology Category

• Machine Learning: Kernel Methods
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Combinatorial Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Decision trees is a fundamental tool in machine learning for representing, classifying, and generalizing data. It is desirable to construct ``small'' decision trees, by minimizing either the size (s) or the depth (d) of the decision tree (DT). Recently, the parameterized complexity of Decision Tree Learning has attracted a lot of attention. We consider a generalization of Decision Tree Learning where given a classification instance E and an integer t, the task is to find a ``small'' DT that disagrees with E in at most t examples. We consider two problems: DTSO and DTDO, where the goal is to construct a DT minimizing s and d, respectively. We first establish that both DTSO and DTDO are W[1]-hard when parameterized by s+y and d+y, respectively, where y is the maximum number of features in which two differently labeled examples can differ. We complement this result by showing that these problems become FPT if we include the parameter t. We also consider the kernelization complexity of these problems and establish several positive and negative results for both DTSO and DTDO.

Problem

• Parameterized complexity of learning small decision trees with few outliers
• Hardness of minimizing decision tree size and depth with feature differences
• Fixed-parameter tractability when including outlier count parameter

Innovation

• Parameterized complexity for decision tree learning
• FPT with parameter t for DTSO and DTDO
• Kernelization analysis for decision tree problems