A Mechanistic Study of Tabular Foundation Models

2026-05-20

AI Summary

This study investigates whether tabular foundation models of different architectures implement identical contextual algorithms, the origins of permutation invariance, and their robustness properties. Through causal interventions, mechanistic interpretability analyses, positional parameter ablations, and targeted adversarial perturbations—such as hub and rank attacks—the work reveals for the first time that distinct model families employ fundamentally different similarity readout mechanisms. It precisely attributes permutation invariance to specific positional parameters, enabling a transition from approximate to exact invariance. The approach accurately reproduces failure modes in model predictions and significantly outperforms retraining baselines, while also demonstrating that representation collapse is not a critical issue in practical settings.

Abstract

Tabular foundation models with different architectures converge in accuracy across a range of classification and regression tasks. This raises questions a leaderboard cannot answer: (i) whether the models execute the same in-context algorithm, (ii) where row, column, and class-permutation invariances originate, and (iii) how robust they are under perturbations engineered against the inferred mechanism. We characterize all three. The model families realize qualitatively distinct similarity-based readouts: from an attention-weighted vote over context labels to a class-conditional mean readout, each confirmed by causal intervention. We find that the representation collapse highlighted in prior work is not a practical concern for them. Each model's permutation invariances trace to specific positional parameters whose removal preserves accuracy and makes approximate invariance exact. Perturbations engineered against each readout reproduce predicted failure modes; hub and rank attacks isolate them from refit baselines. Together these results give a mechanistic account of contemporary tabular foundation models and identify which inductive biases govern both their accuracy and characteristic failures.

Problem

• tabular foundation models
• in-context learning
• permutation invariance
• mechanistic interpretability
• robustness

Innovation

• tabular foundation models
• mechanistic interpretability
• in-context learning
• permutation invariance
• causal intervention