Eliciting Numerical Predictive Distributions of LLMs Without Autoregression

2026-03-03

AI Summary

This work addresses the inefficiency of large language models (LLMs) in regression tasks, where autoregressive decoding for numerical outputs incurs high computational costs and hinders effective modeling of continuous predictive distributions. The study demonstrates for the first time that LLMs inherently encode rich information about numerical uncertainty within their internal representations. Building on this insight, the authors propose a lightweight, non-autoregressive approach that trains regression probes to directly extract statistical functionals—such as means and quantiles—from model embeddings. This method bypasses explicit sampling or autoregressive generation, substantially reducing inference overhead while accurately recovering key distributional statistics. The approach establishes a new paradigm for uncertainty-aware numerical prediction with LLMs.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models (LLMs) have recently been successfully applied to regression tasks -- such as time series forecasting and tabular prediction -- by leveraging their in-context learning abilities. However, their autoregressive decoding process may be ill-suited to continuous-valued outputs, where obtaining predictive distributions over numerical targets requires repeated sampling, leading to high computational cost and inference time. In this work, we investigate whether distributional properties of LLM predictions can be recovered without explicit autoregressive generation. To this end, we study a set of regression probes trained to predict statistical functionals (e.g., mean, median, quantiles) of the LLM's numerical output distribution directly from its internal representations. Our results suggest that LLM embeddings carry informative signals about summary statistics of their predictive distributions, including the numerical uncertainty. This investigation opens up new questions about how LLMs internally encode uncertainty in numerical tasks, and about the feasibility of lightweight alternatives to sampling-based approaches for uncertainty-aware numerical predictions.

Problem

• Large Language Models
• regression
• predictive distributions
• autoregressive decoding
• numerical uncertainty

Innovation

• non-autoregressive prediction
• regression probes
• predictive uncertainty
• statistical functionals
• LLM internal representations