On Finetuning Tabular Foundation Models

2025-06-10

AI Summary

This work addresses the lack of understanding regarding fine-tuning mechanisms and systematic adaptation strategies for tabular foundation models (e.g., TabPFNv2). We present the first in-depth study on parameter-efficient fine-tuning and internal representation evolution in such models. Methodologically, we integrate full-parameter fine-tuning, attention visualization, in-context learning analysis, and cross-distribution generalization evaluation. Our key findings reveal that fine-tuning improves target similarity estimation by calibrating query-key dot products, thereby enhancing context sample weighting accuracy; moreover, fine-tuned models exhibit an intrinsic retrieval-augmented behavior. On IID academic tabular benchmarks with 50K samples, our approach achieves state-of-the-art performance. Empirical analysis confirms full fine-tuning as the optimal trade-off between time efficiency and predictive accuracy, significantly boosting both classification accuracy and contextual utilization efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Foundation models are an emerging research direction in tabular deep learning. Notably, TabPFNv2 recently claimed superior performance over traditional GBDT-based methods on small-scale datasets using an in-context learning paradigm, which does not adapt model parameters to target datasets. However, the optimal finetuning approach for adapting tabular foundational models, and how this adaptation reshapes their internal mechanisms, remains underexplored. While prior works studied finetuning for earlier foundational models, inconsistent findings and TabPFNv2's unique architecture necessitate fresh investigation. To address these questions, we first systematically evaluate various finetuning strategies on diverse datasets. Our findings establish full finetuning as the most practical solution for TabPFNv2 in terms of time-efficiency and effectiveness. We then investigate how finetuning alters TabPFNv2's inner mechanisms, drawing an analogy to retrieval-augmented models. We reveal that the success of finetuning stems from the fact that after gradient-based adaptation, the dot products of the query-representations of test objects and the key-representations of in-context training objects more accurately reflect their target similarity. This improved similarity allows finetuned TabPFNv2 to better approximate target dependency by appropriately weighting relevant in-context samples, improving the retrieval-based prediction logic. From the practical perspective, we managed to finetune TabPFNv2 on datasets with up to 50K objects, observing performance improvements on almost all tasks. More precisely, on academic datasets with I.I.D. splits, finetuning allows TabPFNv2 to achieve state-of-the-art results, while on datasets with gradual temporal shifts and rich feature sets, TabPFNv2 is less stable and prior methods remain better.

Problem

• Optimal finetuning approach for tabular foundation models
• How finetuning reshapes internal mechanisms of TabPFNv2
• Performance impact of finetuning on diverse tabular datasets

Innovation

• Full finetuning optimizes TabPFNv2 effectively
• Finetuning enhances retrieval-based prediction logic
• TabPFNv2 achieves state-of-the-art post-finetuning