Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models

2026-04-14

AI Summary

This work addresses the limitation of existing generalized additive models (GAMs) in automatically identifying higher-order or context-dependent feature interactions, which often rely on heuristic approaches and consequently miss important interaction effects. The authors propose TabDistill, a novel method that leverages a tabular foundation model as a data-driven interaction discovery engine. Specifically, a high-capacity tabular foundation model is first trained, and then post-hoc interaction attribution techniques are applied to extract statistically significant feature interactions. These discovered interactions are subsequently incorporated as explicit terms into a GAM. This approach effectively transfers knowledge from a black-box model to an interpretable GAM framework, yielding substantial improvements in predictive performance across multiple tasks while preserving model interpretability.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Reasoning under Uncertainty: Relational Probabilistic Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Identifying meaningful feature interactions is a central challenge in building accurate and interpretable models for tabular data. Generalized additive models (GAMs) have shown great success at modeling tabular data, but often rely on heuristic procedures to select interactions, potentially missing higher-order or context-dependent effects. To meet this challenge, we propose TabDistill, a method that leverages tabular foundation models and post-hoc distillation methods. Our key intuition is that tabular foundation models implicitly learn rich, adaptive feature dependencies through large-scale representation learning. Given a dataset, TabDistill first fits a tabular foundation model to the dataset, and then applies a post-hoc interaction attribution method to extract salient feature interactions from it. We evaluate these interactions by then using them as terms in a GAM. Across tasks, we find that interactions identified by TabDistill lead to consistent improvements in downstream GAMs' predictive performance. Our results suggest that tabular foundation models can serve as effective, data-driven guides for interaction discovery, bridging high-capacity models and interpretable additive frameworks.

Problem

• feature interactions
• generalized additive models
• tabular data
• interpretability
• interaction selection

Innovation

• feature interactions
• generalized additive models
• tabular foundation models
• model distillation
• interpretability