LassoFlexNet: Flexible Neural Architecture for Tabular Data

2026-03-20

AI Summary

Deep neural networks often underperform tree-based models on tabular data due to irrelevant features, feature heterogeneity, and local irregularities. To address this, this work proposes the LassoFlexNet architecture, which enhances model expressivity through five inductive biases: it employs per-feature embeddings to separately capture linear and nonlinear marginal contributions, integrates a grouped Lasso penalty for sparse variable selection, and introduces a novel Sequential Hierarchical Proximal Adaptive Gradient optimizer incorporating exponential moving averages (EMA). This design effectively breaks undesirable rotational invariance inherent in standard neural networks. Evaluated across 52 benchmark datasets, LassoFlexNet matches or surpasses leading tree-based models, achieving up to a 10% relative performance gain while maintaining strong interpretability.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite their dominance in vision and language, deep neural networks often underperform relative to tree-based models on tabular data. To bridge this gap, we incorporate five key inductive biases into deep learning: robustness to irrelevant features, axis alignment, localized irregularities, feature heterogeneity, and training stability. We propose \emph{LassoFlexNet}, an architecture that evaluates the linear and nonlinear marginal contribution of each input via Per-Feature Embeddings, and sparsely selects relevant variables using a Tied Group Lasso mechanism. Because these components introduce optimization challenges that destabilize standard proximal methods, we develop a \emph{Sequential Hierarchical Proximal Adaptive Gradient optimizer with exponential moving averages (EMA)} to ensure stable convergence. Across $52$ datasets from three benchmarks, LassoFlexNet matches or outperforms leading tree-based models, achieving up to a $10$\% relative gain, while maintaining Lasso-like interpretability. We substantiate these empirical results with ablation studies and theoretical proofs confirming the architecture's enhanced expressivity and structural breaking of undesired rotational invariance.

Problem

• tabular data
• deep neural networks
• tree-based models
• inductive biases
• interpretability

Innovation

• LassoFlexNet
• Tied Group Lasso
• Per-Feature Embeddings
• Sequential Hierarchical Proximal Optimization
• Tabular Data