Closing the gap on tabular data with Fourier and Implicit Categorical Features

2026-02-26

AI Summary

Deep learning models often underperform tree-based methods on tabular data due to difficulties in capturing nonlinear interactions involving categorical features. This work proposes a novel paradigm that first identifies implicit categorical features strongly correlated with the target variable through statistical analysis, then integrates Learned Fourier representations to mitigate deep networks’ tendency toward overly smooth solutions. By enhancing the model’s capacity to capture complex feature interactions, the proposed approach achieves significant performance gains across multiple standard tabular benchmarks, matching or even surpassing the accuracy of XGBoost. This study thus offers an effective pathway for advancing deep learning applications in tabular data domains.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While Deep Learning has demonstrated impressive results in applications on various data types, it continues to lag behind tree-based methods when applied to tabular data, often referred to as the last "unconquered castle" for neural networks. We hypothesize that a significant advantage of tree-based methods lies in their intrinsic capability to model and exploit non-linear interactions induced by features with categorical characteristics. In contrast, neural-based methods exhibit biases toward uniform numerical processing of features and smooth solutions, making it challenging for them to effectively leverage such patterns. We address this performance gap by using statistical-based feature processing techniques to identify features that are strongly correlated with the target once discretized. We further mitigate the bias of deep models for overly-smooth solutions, a bias that does not align with the inherent properties of the data, using Learned Fourier. We show that our proposed feature preprocessing significantly boosts the performance of deep learning models and enables them to achieve a performance that closely matches or surpasses XGBoost on a comprehensive tabular data benchmark.

Problem

• tabular data
• deep learning
• categorical features
• non-linear interactions
• tree-based methods

Innovation

• tabular data
• categorical features
• Learned Fourier
• feature discretization
• deep learning