Distilling Tabular Foundation Models for Structured Health Data

2026-05-18

AI Summary

Although tabular foundation models demonstrate strong performance on medical tasks, their high inference costs and stringent deployment requirements hinder practical adoption. This work proposes a context-leakage-resistant knowledge distillation approach that transfers knowledge from large tabular foundation models to lightweight student models via a hierarchical leave-one-out strategy to generate teacher pseudo-labels. The method substantially reduces inference overhead while preserving model calibration and fairness. Experiments across 19 medical datasets show that the student models retain at least 90% of the teacher’s AUC on average, with some scenarios even yielding superior performance, and achieve over a 26× speedup in CPU-based inference. Additionally, the study finds that ensembling multiple teachers does not consistently improve results.

Abstract

Tabular foundation models (TFMs) achieve strong performance on health datasets, but their inference cost and infrastructure requirements limit practical use. We study whether their predictive behavior can be transferred to lightweight tabular models through knowledge distillation. Since in-context TFMs condition on the training set at inference time, naive distillation can introduce context leakage; we address this with stratified out-of-fold teacher labeling. Across $19$ healthcare datasets, $6$ TFM teachers, $4$ student families, and several multi-teacher ensembles, we find that distilled students retain at least $90\%$ of teacher AUC, outperforming teachers in some cases, while running at least $26\times$ faster on CPU and preserving calibration and fairness critical for health applications. Moreover, multi-teacher averaging does not consistently improve over the best single teacher. Leakage-aware distillation is thus a viable route for bringing TFM-quality predictions into inference-constrained health settings.

Problem

• Tabular Foundation Models
• Knowledge Distillation
• Healthcare Data
• Inference Efficiency
• Context Leakage

Innovation

• knowledge distillation
• tabular foundation models
• context leakage
• healthcare AI
• model compression