Is Deep Learning finally better than Decision Trees on Tabular Data?

2024-02-06

AI Summary

This study systematically evaluates whether deep learning models can surpass gradient-boosted decision trees (GBDTs) in tabular classification, particularly under low-data regimes. We propose the first evaluation framework that categorizes and comparatively benchmarks ten state-of-the-art models according to their learning paradigms: meta-learning, large language model (LLM)-based, specialized neural networks, and AutoML. Empirical results show that meta-learning-based models significantly outperform GBDTs in small-sample settings; specialized neural networks consistently exceed LLM-based approaches in overall accuracy; and while AutoML achieves the highest predictive performance, it incurs the greatest computational cost. Our key contributions are twofold: (1) the first empirical demonstration of meta-learning’s superiority for few-shot tabular learning, and (2) the establishment of a reproducible, extensible benchmark for paradigm-level model evaluation on tabular data.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Tabular data is a ubiquitous data modality due to its versatility and ease of use in many real-world applications. The predominant heuristics for handling classification tasks on tabular data rely on classical machine learning techniques, as the superiority of deep learning models has not yet been demonstrated. This raises the question of whether new deep learning paradigms can surpass classical approaches. Recent studies on tabular data offer a unique perspective on the limitations of neural networks in this domain and highlight the superiority of gradient boosted decision trees (GBDTs) in terms of scalability and robustness across various datasets. However, novel foundation models have not been thoroughly assessed regarding quality or fairly compared to existing methods for tabular classification. Our study categorizes ten state-of-the-art neural models based on their underlying learning paradigm, demonstrating specifically that meta-learned foundation models outperform GBDTs in small data regimes. Although dataset-specific neural networks generally outperform LLM-based tabular classifiers, they are surpassed by an AutoML library which exhibits the best performance but at the cost of higher computational demands.

Problem

• Deep learning vs. decision trees
• Tabular data classification
• Performance comparison

Innovation

• Meta-learned foundation models
• Outperform GBDTs
• Small data regimes