Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue

2026-05-01

AI Summary

This study investigates how speakers make cost-sensitive choices among multiple alternative utterances in dialogue. To this end, it proposes modeling utterance production as a probabilistic selection from contextually grounded sets of alternatives, distinguishing between goal-directed alternatives that realize a fixed communicative intent and goal-agnostic alternatives that are merely contextually plausible. Leveraging language models to generate both types of alternatives, the work introduces a selection mechanism centered on minimizing surprisal with respect to goal-directed alternatives and develops a unified speaker–listener pressure analysis framework. Experimental results demonstrate that this mechanism consistently and substantially outperforms baselines such as uniform information density or length-based cost models across diverse conditions, offering the most robust and accurate predictions of actual human utterance choices.

Abstract

We model utterance production as probabilistic cost-sensitive choice over contextual alternatives, using information-theoretic notions of cost. We distinguish between goal-directed alternatives that realise a fixed communicative intent and goal-agnostic alternatives defined only by contextual plausibility, allowing us to derive speaker- and listener-oriented interpretations of different cost measures. We present a procedure to generate both types of alternative sets using language models. Analysing production choices in open-ended dialogue under both deterministic and probabilistic cost minimisation, we find that surprisal minimisation relative to goal-directed alternatives provides the strongest predictive account under both analyses. By contrast, uniform information density and length-based costs exhibit weaker and less consistent predictive power across conditions. More broadly, our study suggests that alternative-conditioned optimisation with LM-generated alternatives provides a principled framework for studying speaker and listener pressures in naturalistic language production.

Problem

• surprisal minimisation
• goal-directed alternatives
• utterance production
• information-theoretic cost
• dialogue

Innovation

• surprisal minimisation
• goal-directed alternatives
• cost-sensitive choice
• language models
• information-theoretic cost