HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

2026-06-25

AI Summary

Current large language model–driven scientific visualization systems overemphasize autonomy, thereby diminishing human analysts’ control and system transparency. This work proposes a human-in-the-loop, plan-first multi-agent architecture that integrates mixed-initiative interaction—supporting both natural language input and direct manipulation of visualizations—with step-level provenance tracking, sandboxed execution, and test-time learning mechanisms to establish a human-centered collaborative visualization paradigm. User studies demonstrate that the system significantly improves task completion rates, perceived control, and process transparency across users with varying levels of expertise, while also revealing a trade-off between execution efficiency and the degree of human oversight required.

Abstract

Large language model (LLM) agents enable natural language interaction for scientific visualization (SciVis). Still, prior systems have essentially prioritized autonomy over human analytical control, thereby limiting transparency and human oversight. We present HiLSVA, a human-in-the-loop agentic system that supports mixed-initiative SciVis workflows. HiLSVA integrates a plan-first multi-agent architecture with explicit human oversight, stepwise provenance tracking, and learn-at-test-time adaptation from user feedback. The system supports fluid handoff between humans and agents through both natural language and direct manipulation of visualizations, while sandboxed execution ensures safe, reproducible workflows. In doing so, HiLSVA reframes agentic SciVis as a collaborative process that augments, rather than replaces, human analytical reasoning. We evaluate HiLSVA through representative case studies and a controlled user study with twelve participants of varying expertise across multiple autonomy settings. Results show that mixed-initiative interaction improves task completion, user control, and workflow transparency across different levels of user expertise, while revealing a tradeoff between execution efficiency and human oversight. These findings highlight the importance of human-centered design in agentic SciVis and guide the development of future collaborative visualization systems. We encourage readers to explore our demo video, case studies, and source code at https://hilsva.github.io/.

Problem

• scientific visualization
• human-in-the-loop
• LLM agents
• mixed-initiative interaction
• human oversight

Innovation

• human-in-the-loop
• mixed-initiative interaction
• plan-first multi-agent architecture
• learn-at-test-time adaptation
• provenance tracking