Uncertainty-Aware Generation and Decision-Making Under Ambiguity

2026-06-29

AI Summary

Large language models struggle to effectively model uncertainty in highly subjective and ambiguous tasks, often producing outputs with insufficient reliability. This work proposes a novel uncertainty-aware generation and decision-making framework that systematically integrates Bayesian decision theory with conformal prediction, tailored for applications such as tutoring and automated peer review. The approach incorporates risk-averse strategies to govern the generation process while providing formal guarantees of reliability. Experimental results demonstrate that the proposed algorithm significantly enhances generation utility across most settings. Notably, the Bayesian formulation consistently outperforms purely risk-averse strategies, which tend to produce overly generic outputs in high-ambiguity scenarios, thereby degrading performance.

Abstract

With rapidly improving capabilities, Large Language Models (LLMs) are increasingly used in many complex real-world tasks. Beyond requiring in-depth knowledge and reasoning skills, many of these tasks exhibit a high degree of subjectivity and require that the outputs of the model can be trusted. While a lot of progress has been made to train better models, decision-making algorithms have received less attention. In this work, we present and evaluate various uncertainty-aware decision-making algorithms based on Bayesian decision theory and risk-averse decision making on the tasks of tutoring and automatic peer reviewing. Concretely, we take uncertainty over tutoring strategies and review scores into account when generating a tutor response or review and use conformal prediction to provide guarantees over strategy and score. We find empirically that these algorithms can improve the utility of the generations but need to be carefully implemented when ambiguity is high. For example, risk-averse rules can degrade performance by optimizing for generic outputs, while Bayesian methods tend to perform better. Our work uses techniques from decision theory to improve LLM-based decision-making and outlines open challenges for the community.

Problem

• uncertainty-aware
• decision-making
• ambiguity
• Large Language Models
• Bayesian decision theory

Innovation

• uncertainty-aware generation
• Bayesian decision theory
• conformal prediction
• risk-averse decision making
• large language models