Iterative Feature Exclusion Ranking for Deep Tabular Learning

2024-12-21

arXiv.org

AI Summary

Existing deep tabular models employ unidimensional feature selection mechanisms, which struggle to capture context-dependent importance and feature interactions, while remaining vulnerable to bias from high-influence features and limited attention generalization. To address these limitations, we propose an iterative feature ablation–driven importance re-ranking paradigm: features are sequentially masked to explicitly model global dependencies and local interactions via dynamic attention score analysis; multi-round attention aggregation coupled with end-to-end gradient optimization further enhances robustness in importance estimation. Evaluated on four standard tabular benchmarks, our method achieves significant improvements over state-of-the-art approaches in both feature ranking accuracy and downstream classification performance. Notably, this work is the first to systematically integrate iterative ablation into explainability modeling for deep tabular learning.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Tabular data is a common format for storing information in rows and columns to represent data entries and their features. Although deep neural networks have become the main approach for modeling a wide range of domains including computer vision and NLP, many of them are not well-suited for tabular data. Recently, a few deep learning models have been proposed for deep tabular learning, featuring an internal feature selection mechanism with end-to-end gradient-based optimization. However, their feature selection mechanisms are unidimensional, and hence fail to account for the contextual dependence of feature importance, potentially overlooking crucial interactions that govern complex tasks. In addition, they overlook the bias of high-impact features and the risk associated with the limitations of attention generalization. To address this limitation, this study proposes a novel iterative feature exclusion module that enhances the feature importance ranking in tabular data. The proposed module iteratively excludes each feature from the input data and computes the attention scores, which represent the impact of the features on the prediction. By aggregating the attention scores from each iteration, the proposed module generates a refined representation of feature importance that captures both global and local interactions between features. The effectiveness of the proposed module is evaluated on four public datasets. The results demonstrate that the proposed module consistently outperforms state-of-the-art methods and baseline models in feature ranking and classification tasks. The code is publicly available at https://github.com/abaraka2020/Iterative-Feature-Exclusion-Ranking-Module and https://github.com/mohalim/IFENet

Problem

• Enhances feature importance ranking in tabular data
• Addresses limitations of unidimensional feature selection mechanisms
• Captures global and local interactions between features

Innovation

• Iterative feature exclusion for ranking importance
• Aggregates attention scores across multiple iterations
• Captures global and local feature interactions