Connecting the Dots: Surfacing Structure in Documents through AI-Generated Cross-Modal Links

2026-02-18

AI Summary

This work addresses the challenge of fragmented information in complex scientific documents, where content is dispersed across textual passages and visual elements such as figures and tables, imposing high cognitive load on readers. To mitigate this, the paper proposes a fine-grained information integration framework that achieves, for the first time, systematic and granular cross-modal alignment between textual and visual components in scientific papers. Leveraging an AI-driven alignment algorithm and an interactive visualization interface—featuring clickable figures, synchronized text highlighting, and a persistent reference panel—the framework enables seamless navigation and integration of dispersed information. User studies demonstrate that the tool significantly improves reading comprehension accuracy without increasing task completion time or cognitive burden.

Technology Category

• Natural Language Processing: Information Extraction
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Interdisciplinary science discovery with web data mining
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Understanding information-dense documents like recipes and scientific papers requires readers to find, interpret, and connect details scattered across text, figures, tables, and other visual elements. These documents are often long and filled with specialized terminology, hindering the ability to locate relevant information or piece together related ideas. Existing tools offer limited support for synthesizing information across media types. As a result, understanding complex material remains cognitively demanding. This paper presents a framework for fine-grained integration of information in complex documents. We instantiate the framework in an augmented reading interface, which populates a scientific paper with clickable points on figures, interactive highlights in the body text, and a persistent reference panel for accessing consolidated details without manual scrolling. In a controlled between-subjects study, we find that participants who read the paper with our tool achieved significantly higher scores on a reading quiz without evidence of increased time to completion or cognitive load. Fine-grained integration provides a systematic way of revealing relationships within a document, supporting engagement with complex, information-dense materials.

Problem

• information-dense documents
• cross-modal links
• document understanding
• cognitive load
• information integration

Innovation

• cross-modal linking
• fine-grained integration
• augmented reading interface
• AI-generated document structure
• information synthesis