A Multimodal Reasoning Typology for Grounding Chart-Image Coherence in Science Communication

2026-07-06

AI Summary

In scientific communication, charts and images are often presented together, yet their multimodal coherence remains poorly characterized, potentially leading to interpretive gaps for readers. Addressing this issue, this study draws on pragmatic grounding theory to conceptualize chart–image–text as an integrated multimodal unit. Through qualitative coding of 32 chart–image pairs from 79 traumatic brain injury research papers, the authors propose a typology of multimodal inference—R1 through R5. Evaluated on an additional 25 pairs, this framework effectively predicts both convergence and divergence in comprehension between expert and non-expert audiences, revealing that contextual knowledge, rather than visual content per se, underpins perceived coherence. The work thus offers a theoretically grounded and actionable foundation for designing and objectively evaluating scientific visualizations.

Abstract

Charts and images appear together throughout scientific publications, yet most computational work does not characterize their coherence. We argue that a chart, its accompanying image, and the caption that links them form a multimodal unit, and that the inferential work required to read it varies systematically. To capture this variation, we develop a typology of reasoning gaps, R1 through R5, that characterizes how chart, image, and text jointly convey a scientific claim, and the interpretive work this demands of the reader. Some pairs restate the same data, while in other pairs, charts are used to quantify a structure the image localizes, project image content onto an external variable, audit an image-based claim, or jointly construct a frame that neither panel can establish alone. The typology is anchored in the grounding theory of communication and was derived bottom-up, with a neuroscience expert, from a corpus of 79 traumatic brain injury papers and 32 chart-image pairs. Crucially, the levels provide a systematic mechanism for identifying where grounding succeeds or breaks down, rather than leaving it to subjective inference. We show this in a study in which a domain expert and three non-experts judge vision-language model (VLM) descriptions of 25 pairs: the level predicts where their judgments align and where they diverge, isolating the points at which contextual knowledge, not the figure, carries coherence. This typology thus offers figure designers a systematic way to balance text against chart-image pairs, bridging the expert-to-non-expert divide in reading a scientific takeaway.

Problem

• multimodal reasoning
• chart-image coherence
• scientific communication
• grounding theory
• reasoning gaps

Innovation

• multimodal reasoning
• chart-image coherence
• grounding theory
• reasoning typology
• vision-language models