When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

2026-05-27

AI Summary

This study addresses the lack of systematic evaluation regarding whether current vision-language models can appropriately structure information—such as distinguishing topic from focus—in visual question answering according to discourse context. Introducing information structure theory into this domain for the first time, the work leverages the fixed syntactic positions of topic and focus in Hungarian, combined with cross-linguistic pragmatic analysis and fine-grained annotation, to conduct comparative experiments between six state-of-the-art models and human participants. The findings reveal that while models exhibit some capacity to encode information structure, they tend to rely on rigid templates under the multifaceted pressures of discourse status, grammatical roles, and definiteness, displaying markedly less strategic flexibility and diversity than humans.

Abstract

Vision-language models (VLMs) are increasingly evaluated for whether they identify the right visual content, but little is known about whether they express such content in a discourse-appropriate form. We address this research gap using information structure (IS), testing whether VLMs distinguish discourse-old Topics from discourse-new Foci in visually grounded question answering. We exploit Hungarian, a language in which Topic and Focus map onto dedicated syntactic positions, making IS choices observable in text. Comparing six VLMs with human participants, we find that models produce IS-relevant constructions, but over-regularise this sensitivity. Under the interacting pressures of discourse status, grammatical role (preference for subject Topics) and definiteness (preference for indefinite Foci), humans choose variable strategies for IS realisation. VLMs, by contrast, collapse onto narrow response templates, resembling mode collapse (Kirk et al., 2024). Our findings suggest that VLM evaluation should look beyond content accuracy to how content is packaged for the discourse.

Problem

• vision-language models
• information structure
• discourse
• topic-focus
• language generation

Innovation

• information structure
• vision-language models
• discourse appropriateness
• mode collapse
• Hungarian syntax