TabPATE: Differentially Private Tabular In-Context Learning Without Public Data

2026-06-30

AI Summary

This work addresses the vulnerability of tabular data to membership inference attacks in in-context learning, which can lead to the leakage of private records. To mitigate this risk, the authors propose TabPATE—the first differentially private PATE framework that operates without requiring access to public data drawn from the same distribution as the private dataset. TabPATE partitions the private contextual data to train multiple teacher models, generates synthetic queries using either feature ranges or mildly privatized marginal distributions, and aggregates teacher predictions via a differentially private mechanism to produce labeled student contexts for secure inference. Experimental results demonstrate that TabPATE achieves competitive model utility across multiple tabular benchmarks while reducing the success rate of membership inference attacks to near-random levels.

Abstract

Tabular foundation models enable accurate in-context learning (ICL) from small labeled datasets, but the private records placed in context can leak through model predictions. We first show that even basic membership inference attacks succeed against tabular ICL, motivating formal privacy protection. We then introduce TabPATE, a differentially private PATE-style defense for tabular ICL that does not require public in-distribution data. TabPATE partitions the private context across teacher models, privately aggregates their labels on synthetic tabular queries, and releases the resulting labeled queries as a student context. Because tabular features are bounded and relatively low-dimensional, useful queries can be generated from feature ranges alone or from lightly privatized marginals. Across tabular benchmarks, TabPATE preserves competitive utility while reducing membership inference to near-random success, providing a practical path to private tabular ICL without public data.

Problem

• differential privacy
• tabular data
• in-context learning
• privacy leakage
• membership inference

Innovation

• differential privacy
• tabular in-context learning
• PATE
• synthetic query generation
• membership inference