KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Models

2026-06-29

AI Summary

This work addresses the limitations of tabular foundation models in small-scale specialized domains, where performance is hindered by data scarcity, high dimensionality, distributional shifts, and the absence of mechanisms to effectively incorporate structured priors such as knowledge graphs. To overcome these challenges, the authors propose a knowledge-guided fine-tuning approach that, for the first time, integrates structural attention priors derived from knowledge graphs with low-rank adaptation (LoRA) to efficiently inject domain knowledge during fine-tuning. Experiments on lightweight tabular models—including TabPFN and TabICL—demonstrate that the method substantially outperforms standard fine-tuning on specialized tasks, while yielding only marginal gains on general tasks. These results validate the efficacy of structured knowledge injection and reveal that continued fine-tuning may lead to the collapse of pre-trained knowledge.

Abstract

Tabular foundation models have advanced deep learning for tabular data by delivering strong default performance across many small and medium tasks. Yet in niche domains, where data is scarce, high-dimensional, and shifted from the pretraining distribution, they may still fail to outperform carefully designed domain-specific methods. Many such domains also provide curated relational knowledge in the form of knowledge graphs and knowledge banks, but how to use this knowledge to improve and steer \textit{small} specialist tabular foundation models remains unclear. We address this problem through \textbf{Know}ledge-informed fine-tuning of \textbf{s}mall \textbf{T}abular \textbf{F}oundation \textbf{M}odels (\modelname). Specifically, we study nanoscale TabPFN- and TabICL-style variants, pretrained under controlled synthetic prior families and adapted using two complementary mechanisms: structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. We show that injecting domain-specific structural knowledge during fine-tuning yields meaningful gains over vanilla variants in specialist settings, whereas gains on general-domain tasks are marginal. We further observe that continual fine-tuning of frontier models can trigger collapse of pretrained knowledge and mechanisms.

Problem

• tabular foundation models
• knowledge graphs
• domain-specific knowledge
• fine-tuning
• data scarcity

Innovation

• knowledge-informed fine-tuning
• tabular foundation models
• knowledge graphs
• low-rank adaptation
• structural attention priors