NLI4VolVis: Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splatting

2025-07-16

AI Summary

Traditional volume visualization relies on manually designed transfer functions and incurs high computational overhead; while existing view synthesis methods improve efficiency, they lack semantic interaction capabilities and usability. This paper introduces the first natural language–driven volume visualization system for non-expert users. Our approach employs a multi-agent large language model (LLM) architecture that integrates vision-language models, multi-view semantic segmentation, and editable 3D Gaussian splatting rendering. It enables open-vocabulary querying, real-time scene editing, and multi-view semantic segmentation. The end-to-end framework significantly lowers the interaction barrier. Extensive case studies and user experiments demonstrate its efficacy, usability, and interactive flexibility. By unifying natural language understanding with geometric and visual reasoning, our system establishes a new paradigm for semantic, low-threshold volume data exploration.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Traditional volume visualization (VolVis) methods, like direct volume rendering, suffer from rigid transfer function designs and high computational costs. Although novel view synthesis approaches enhance rendering efficiency, they require additional learning effort for non-experts and lack support for semantic-level interaction. To bridge this gap, we propose NLI4VolVis, an interactive system that enables users to explore, query, and edit volumetric scenes using natural language. NLI4VolVis integrates multi-view semantic segmentation and vision-language models to extract and understand semantic components in a scene. We introduce a multi-agent large language model architecture equipped with extensive function-calling tools to interpret user intents and execute visualization tasks. The agents leverage external tools and declarative VolVis commands to interact with the VolVis engine powered by 3D editable Gaussians, enabling open-vocabulary object querying, real-time scene editing, best-view selection, and 2D stylization. We validate our system through case studies and a user study, highlighting its improved accessibility and usability in volumetric data exploration. We strongly recommend readers check our case studies, demo video, and source code at https://nli4volvis.github.io/.

Problem

• Enables natural language interaction for volume visualization
• Overcomes rigid transfer function designs and high computational costs
• Supports semantic-level interaction and real-time scene editing

Innovation

• Multi-agent LLM architecture for intent interpretation
• Editable 3D Gaussian splatting for real-time editing
• Vision-language models for semantic scene understanding