JoLT: Joint Probabilistic Predictions on Tabular Data Using LLMs

2025-02-17

AI Summary

Addressing challenges in joint probabilistic modeling of tabular data—including multi-target prediction, heterogeneous variable types (classification/regression), and robustness to missing values—this paper proposes JoLT, the first end-to-end joint probability modeling framework leveraging large language model (LLM) in-context learning. JoLT requires no model training, data preprocessing, or explicit imputation; instead, it achieves direct joint distribution estimation via structured-to-textual data encoding and probabilistic prompting. Its key contributions are: (1) the first native application of LLMs to tabular joint probability modeling; (2) zero-shot automatic handling of missing values and seamless integration of auxiliary textual side information; and (3) unified support for both single- and multi-target settings across classification and regression tasks. Experiments demonstrate that JoLT significantly outperforms state-of-the-art methods in low-data regimes and enables high-fidelity synthetic data reconstruction.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We introduce a simple method for probabilistic predictions on tabular data based on Large Language Models (LLMs) called JoLT (Joint LLM Process for Tabular data). JoLT uses the in-context learning capabilities of LLMs to define joint distributions over tabular data conditioned on user-specified side information about the problem, exploiting the vast repository of latent problem-relevant knowledge encoded in LLMs. JoLT defines joint distributions for multiple target variables with potentially heterogeneous data types without any data conversion, data preprocessing, special handling of missing data, or model training, making it accessible and efficient for practitioners. Our experiments show that JoLT outperforms competitive methods on low-shot single-target and multi-target tabular classification and regression tasks. Furthermore, we show that JoLT can automatically handle missing data and perform data imputation by leveraging textual side information. We argue that due to its simplicity and generality, JoLT is an effective approach for a wide variety of real prediction problems.

Problem

• Probabilistic predictions on tabular data
• Handling missing data automatically
• Outperforming competitive methods in classification

Innovation

• LLM-based joint probabilistic predictions
• Handles heterogeneous data without preprocessing
• Automatic missing data imputation