Table Foundation Models: on knowledge pre-training for tabular learning

2025-05-20

AI Summary

Existing tabular pre-trained models suffer from weak semantic understanding, high computational overhead, poor reusability, and tight coupling with downstream architectures. To address these challenges, this paper proposes TARTE—the first knowledge-enhanced foundational model for tabular data. Methodologically, TARTE introduces: (1) a lightweight representation paradigm that models semantic associations between column names and cell values via string-based encoding; (2) a plug-and-play, composable, and domain-specializable knowledge-aware pre-training framework, enabling zero- or few-shot fine-tuning and cross-architecture reuse; and (3) large-scale self-supervised pre-training on relational data, jointly optimizing semantic encoding and knowledge-augmented vector learning. Experiments demonstrate that TARTE achieves state-of-the-art accuracy across diverse tabular benchmarks, accelerates inference by 3–5×, reduces fine-tuning parameters by over 90%, and validates the feasibility of efficient, general-purpose, and transferable tabular foundation models.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Knowledge Engineering
• Data Mining & Knowledge Management: Data Compression

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Table foundation models bring high hopes to data science: pre-trained on tabular data to embark knowledge or priors, they should facilitate downstream tasks on tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, e.g., column name. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience of popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot easily be reused or combined with other architectures. Here we introduce TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using the string to capture semantics. Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be fine-tuned or combined with other learners, giving models that push the state-of-the-art prediction performance and improve the prediction/computation performance trade-off. Specialized to a task or a domain, TARTE gives domain-specific representations that facilitate further learning. Our study demonstrates an effective approach to knowledge pre-training for tabular learning.

Problem

• Pre-training models for tabular data lack convenience and reusability
• Numerical entries require context-aware semantics for accurate interpretation
• Existing models need fine-tuning and incur high computation costs

Innovation

• Pre-trained neural networks model column names and entries
• TARTE transforms tables to knowledge-enhanced vector representations
• TARTE enables fine-tuning and combination with other learners