TACTICL: Task-Aware Compression of Tabular ICL Models

2026-08-11

AI Summary

This work addresses the high inference cost of foundation models in tabular data in-context learning by proposing a task-aware compression framework that, for the first time, integrates structured pruning with lightweight task adapters to substantially reduce model size while preserving in-context learning capabilities. The approach jointly prunes redundant Transformer layers and introduces learnable adapter modules, enabling efficient and automated task-specific compression. Extensive experiments across 47 benchmark datasets demonstrate that the framework can safely remove up to 85% of model layers with minimal performance degradation and maintains robustness under distribution shifts.

Abstract

The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression

Problem

• tabular models
• in-context learning
• model compression
• task-specific adaptation
• foundation models

Innovation

• task-aware compression
• tabular in-context learning
• structured pruning
• lightweight adapters
• foundation model compression