Sparse Oblique Rule Boosting for Simpler Additive Rule Ensembles

2026-09-06

AI Summary

本文通过引入可学习的稀疏线性变换作为规则条件，结合梯度提升方法，解决了传统规则集成模型在缺乏精心设计特征时准确性与解释性之间的矛盾。

Abstract

Small additive ensembles of symbolic rules offer interpretable prediction models. Traditionally, these ensembles use rule conditions based on conjunctions of simple threshold propositions $x \geq t$ on a single input variable $x$ and threshold $t$, resulting geometrically in axis-parallel polytopes as decision regions. While this form ensures a high degree of interpretability for individual rules and can be learned efficiently using the gradient boosting approach, it relies on having access to a curated set of expressive input features so that a small ensemble of axis-parallel regions can describe the target variable well. Absent such features, reaching sufficient accuracy requires increasing the number and complexity of individual rules, which diminishes the interpretability of the model. Here, we extend classical rule ensembles by introducing logical propositions with learnable sparse linear transformations of input variables, i.e., propositions of the form $\mathbf{x}^T\mathbf{w} \geq t$, where $\mathbf{w}$ is a learnable sparse weight vector, enabling decision regions as general polyhedrons with oblique faces. We propose a learning method using gradient boosting based on a weighted logistic regression. Empirical results across 14 regression and classification tasks demonstrate that the proposed method achieves lower model complexity than competitive baselines while maintaining similar or better predictive accuracy. Hence, the approach provides a favorable trade-off between interpretability and accuracy and reduces the reliance on manual feature engineering.

Problem

• sparse oblique rules
• additive rule ensembles
• interpretability
• gradient boosting
• feature engineering

Innovation

• Sparse Linear Transformations
• Oblique Decision Regions
• Gradient Boosting
• Interpretable Models
• Feature Engineering