Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation

2026-07-06

AI Summary

This work addresses negative transfer in tabular foundation models under transfer learning, which arises from context length constraints and source–target distribution shifts. To mitigate this issue, the authors propose TL-ANDI, a novel framework that, for the first time, integrates optimal transport with posterior compatibility to select anchor samples that both cover the target covariate space and align with the target task’s posterior distribution—all within a strict context budget. TL-ANDI further enhances contextual efficiency by combining local label distillation, residual calibration, and context anchoring to construct compact yet informative contexts. Extensive experiments demonstrate that TL-ANDI significantly alleviates negative transfer and consistently improves both transfer performance and context utilization across multiple cross-domain tabular tasks.

Abstract

Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning. However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks. Directly pooling heterogeneous source data can therefore lead to negative transfer. To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs. TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility. The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.

Problem

• Tabular Foundation Models
• Transfer Learning
• Context Constraints
• Distribution Shift
• Negative Transfer

Innovation

• Tabular Foundation Models
• Transfer Learning
• Data Distillation
• Optimal Transport
• Posterior Compatibility