Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

2026-01-12

AI Summary

Current language models struggle to recognize their own uncertainty during interaction and proactively request clarification, limiting human-AI alignment and mutual understanding. This work introduces the first systematic formulation of the referring game as a controllable evaluation platform to explicitly quantify clarification-seeking behavior in vision-language models under conditions of uncertainty. Building upon standard referring expression comprehension tasks, we conduct comparative experiments by incorporating clarification prompts into three mainstream vision-language models. Our results demonstrate that even in simple scenarios, these models fail to accurately perceive their internal uncertainty or generate appropriate clarification requests, revealing a critical limitation in their capacity for interactive understanding.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In human conversation, both interlocutors play an active role in maintaining mutual understanding. When addressees are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar addressee role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a good testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.

Problem

• clarification requests
• model uncertainty
• reference games
• language models
• mutual understanding

Innovation

• reference games
• model uncertainty
• clarification requests
• vision-language models
• interactive alignment