Automatic Semantic Alignment of Flow Pattern Representations for Exploration with Large Language Models

2025-08-08

AI Summary

Traditional flow visualization relies on domain-specific graphical interfaces, imposing high learning barriers; while natural language interaction offers intuitive access, it faces dual challenges in scientific concept recognition and flow structure extraction. To address these, we propose the first annotation-free semantic alignment framework: a denoising autoencoder learns robust representations of streamline segments; a trainable projection layer maps these representations into the embedding space of large language models (LLMs); and a cross-modal attention mechanism enables precise alignment between textual descriptions and underlying flow patterns. Based on this, we design a natural language–driven interactive exploration interface. Case studies with domain experts demonstrate that users can directly query and visualize complex flow structures using plain-language instructions—significantly lowering usability barriers, improving analytical efficiency, and enhancing accessibility for non-expert users.

Technology Category

• Natural Language Processing: (Large) Language Models
• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Natural language understanding for Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Explorative flow visualization allows domain experts to analyze complex flow structures by interactively investigating flow patterns. However, traditional visual interfaces often rely on specialized graphical representations and interactions, which require additional effort to learn and use. Natural language interaction offers a more intuitive alternative, but teaching machines to recognize diverse scientific concepts and extract corresponding structures from flow data poses a significant challenge. In this paper, we introduce an automated framework that aligns flow pattern representations with the semantic space of large language models (LLMs), eliminating the need for manual labeling. Our approach encodes streamline segments using a denoising autoencoder and maps the generated flow pattern representations to LLM embeddings via a projector layer. This alignment empowers semantic matching between textual embeddings and flow representations through an attention mechanism, enabling the extraction of corresponding flow patterns based on textual descriptions. To enhance accessibility, we develop an interactive interface that allows users to query and visualize flow structures using natural language. Through case studies, we demonstrate the effectiveness of our framework in enabling intuitive and intelligent flow exploration.

Problem

• Align flow patterns with LLM semantics for intuitive exploration
• Automate flow pattern recognition without manual labeling
• Enable natural language queries for flow visualization

Innovation

• Automated alignment of flow patterns with LLM semantics
• Denoising autoencoder encodes streamline segments
• Attention mechanism enables semantic matching for queries