Basis Transformers for Multi-Task Tabular Regression

2025-06-07

AI Summary

To address challenges in tabular data—including missing values, noise, heterogeneous structures, variable column counts, and absence of metadata—this paper proposes Basis Transformers, a novel architecture for multi-task tabular regression. Methodologically, it introduces, for the first time, a basis decomposition mechanism into tabular Transformers, explicitly modeling column-name semantics and value-level scale invariance without requiring pretraining or metadata. The design integrates column-aware basis vector decomposition, a lightweight attention module, and task-shared parametric representations. Evaluated on 34 regression tasks from OpenML-CTR23, Basis Transformers achieves a median R² improvement of 0.338 over strong baselines, attains the lowest standard deviation across tasks, and uses only one-fifth the parameters of the best-performing baseline. Notably, with random initialization alone, it surpasses fine-tuned large language models—demonstrating superior parameter efficiency, robustness, and generalization without reliance on external knowledge or costly adaptation.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Metareasoning and Metaheuristics
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Dealing with tabular data is challenging due to partial information, noise, and heterogeneous structure. Existing techniques often struggle to simultaneously address key aspects of tabular data such as textual information, a variable number of columns, and unseen data without metadata besides column names. We propose a novel architecture, extit{basis transformers}, specifically designed to tackle these challenges while respecting inherent invariances in tabular data, including hierarchical structure and the representation of numeric values. We evaluate our design on a multi-task tabular regression benchmark, achieving an improvement of 0.338 in the median $R^2$ score and the lowest standard deviation across 34 tasks from the OpenML-CTR23 benchmark. Furthermore, our model has five times fewer parameters than the best-performing baseline and surpasses pretrained large language model baselines -- even when initialized from randomized weights.

Problem

• Handling partial noisy heterogeneous tabular data
• Addressing textual variable-column unseen tabular data
• Respecting hierarchical numeric invariances in tables

Innovation

• Basis transformers for tabular regression
• Handles textual and numeric data
• Fewer parameters than baselines