TabICLv2: A better, faster, scalable, and open tabular foundation model

2026-02-11

AI Summary

This work proposes an efficient and scalable tabular foundation model to address the limitations of existing tabular prediction approaches in generalization, efficiency, and scalability. Built upon the Transformer architecture, the model introduces several key innovations: a high-diversity synthetic data generation strategy, a scalable Softmax attention mechanism, and, for the first time in tabular modeling, the adoption of the Muon optimizer. Remarkably, the model achieves strong out-of-the-box generalization without fine-tuning, surpassing the current state-of-the-art method, RealTabPFN-2.5, on both TabArena and TALENT benchmarks. It supports inference on datasets with up to one million samples while consuming less than 50 GB of GPU memory and delivers significantly improved inference speed.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and commit to open research by first releasing inference code and model weights at https://github.com/soda-inria/tabicl, with synthetic data engine and pretraining code to follow.

Problem

• tabular foundation model
• in-context learning
• scalability
• generalization
• large-scale datasets

Innovation

• synthetic data generation
• scalable softmax attention
• in-context learning
• tabular foundation model
• Muon optimizer