Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data

2026-08-28

AI Summary

本文提出了一种无需共享特征的跨域注意力转移学习方法CATTLE，通过Transformer投影权重捕获泛化上下文，实验显示其优于多种现有方法。

Abstract

Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. \mds{This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for $key$, $value$, and $query$ provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for $key$ from the source domain interact with the weight for $query$ in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods.} The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.

Problem

• tabular data
• transfer learning
• heterogeneity
• disjoint domains
• shared features

Innovation

• generalized context learning
• cross-domain attention transfer learning
• transformer projection weights
• disjoint tabular data
• data-agnostic