A Unified Approach to Extract Interpretable Rules from Tree Ensembles via Integer Programming

2024-06-30

AI Summary

To address the interpretability bottleneck of tree ensemble models, this paper proposes an integer programming (IP)-based rule extraction framework that unifies rule learning as a set cover problem with arbitrary loss and regularization terms. Methodologically, it innovatively integrates set partition modeling, rule coverage encoding, and customizable loss/regularization embedding, supporting both classification and regression tasks on tabular and time-series data. Compared to state-of-the-art approaches, it achieves superior performance across diverse benchmark tasks: extracting fewer than 15 high-fidelity rules on average—semantically clear and preserving over 95% of the original ensemble’s accuracy—while guaranteeing theoretical optimality. This work presents the first unified IP formulation for rule extraction from tree ensembles, simultaneously ensuring conciseness, interpretability, and predictive fidelity.

Technology Category

• Machine Learning: Ensemble Methods
• Data Mining & Knowledge Management: Rule Mining & Pattern Mining
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Tree ensembles are very popular machine learning models, known for their effectiveness in supervised classification and regression tasks. Their performance derives from aggregating predictions of multiple decision trees, which are renowned for their interpretability properties. However, tree ensemble models do not reliably exhibit interpretable output. Our work aims to extract an optimized list of rules from a trained tree ensemble, providing the user with a condensed, interpretable model that retains most of the predictive power of the full model. Our approach consists of solving a set partitioning problem formulated through Integer Programming. The proposed method works with either tabular or time series data, for both classification and regression tasks, and its flexible formulation can include any arbitrary loss or regularization functions. Our extensive computational experiments offer statistically significant evidence that our method is competitive with other rule extraction methods in terms of predictive performance and fidelity towards the tree ensemble. Moreover, we empirically show that the proposed method effectively extracts interpretable rules from tree ensemble that are designed for time series data.

Problem

• Tree Ensembles
• Simplification
• Model Interpretability

Innovation

• Integer Programming
• Rule Extraction
• Time Series Data