SOMTab: Set-Order Mamba for Efficient Tabular In-Context Learning

2026-08-27

AI Summary

为提高表格上下文学习效率，SOMTab通过分离表示构建与查询条件检索，并采用Mamba状态空间混合及注意力机制，在保持准确度的同时降低了GPU内存使用和加速了推理过程。

Abstract

Tabular foundation models based on in-context learning have recently emerged as strong alternatives to task-specific model fitting. However, the current performance frontier remains dominated by attention-heavy architectures, where attention is used throughout the modeling pipeline. This raises a natural question: is attention necessary at every stage of tabular in-context learning? We introduce SOMTab, a Set-Order Mamba architecture for efficient tabular in-context learning. SOMTab separates representation construction from query-conditioned retrieval. For row and column representations, it maps unordered table tokens into stable latent slots and applies Mamba-based state-space mixing to construct compact representations. For final prediction, it retains attention-based in-context learning to preserve query-conditioned retrieval from labeled context examples. We further introduce DCH-TailMix, a synthetic prior that combines degree-corrected graph heterogeneity with mixed heavy-tailed regimes to diversify synthetic dependency structures. Across tabular benchmarks, SOMTab approaches the performance of strong Transformer-based tabular foundation models while achieving faster inference and lower GPU memory usage, yielding a favorable efficiency--accuracy trade-off.

Problem

• attention
• tabular in-context learning
• efficiency

Innovation

• Set-Order Mamba
• tabular in-context learning
• state-space mixing
• DCH-TailMix
• efficient representation