In-Context Density Estimation for Tabular Data

2026-08-10

AI Summary

This work addresses the inefficiency of conventional tabular density estimation methods, which require separate model training and extensive hyperparameter tuning for each task. The authors propose ICED, a Transformer-based in-context energy-based density estimator that, after a single pretraining phase on synthetic prior data, directly outputs unnormalized log-densities for arbitrary query points without fine-tuning, sampling, or hyperparameter adjustment. ICED is the first method to unify four distinct tasks—density estimation, out-of-distribution detection, unsupervised anomaly detection, and generative data augmentation—within a single frozen model. Experimental results demonstrate that ICED achieves performance on par with the strongest task-specific baselines across all settings while substantially reducing the computational and engineering overhead associated with multi-task deployment.

Abstract

Density estimation underlies many unsupervised tasks on tabular data such as anomaly detection, out-of-distribution detection, and data augmentation. Although all these problems reduce to questions about where probability mass lies, they are typically solved individually by fitting a separate model to each dataset, with its own hyperparameters and tuning budget. We introduce ICED, an in-context, energy-based density estimator that removes this per-dataset cost. ICED is a transformer-based model pretrained once on a synthetic prior built specifically for density estimation under an objective that fits log-density where it is informative and preserves its ordering elsewhere. In the inference, it reads a dataset as context and returns an unnormalized log-density for any query point in a single forward pass, with no fitting, sampling, or hyperparameter selection. A single frozen ICED model then drives four tasks usually handled by four specialized pipelines: density estimation, out-of-distribution detection, unsupervised anomaly detection, and generative augmentation. Across all four, it is competitive with the strongest task-specific method, while being the only approach that needs no retraining, no tuning, and no labels to move between them. The code is available at https://github.com/gmum/iced.

Problem

• density estimation
• tabular data
• in-context learning
• unsupervised tasks
• out-of-distribution detection

Innovation

• in-context learning
• density estimation
• energy-based model
• tabular data
• transformer