ParaView-MCP: An Autonomous Visualization Agent with Direct Tool Use

2025-05-11

AI Summary

Scientific visualization tools such as ParaView impose high learning and interaction barriers, limiting accessibility for domain scientists. Method: This paper proposes a natural language (NL) and vision-language (VL) driven autonomous visualization agent. We introduce the first Model Context Protocol (MCP)-based architecture that directly integrates multimodal large language models (MLLMs) with ParaView, enabling real-time viewport image observation and interpretation. A closed-loop visualization feedback mechanism supports example-based replication, goal-oriented parameter optimization, and cross-tool orchestration. Contribution/Results: We achieve the first end-to-end NL/VL-driven automation of ParaView operations; significantly reduce user cognitive load and lower entry barriers; and empirically demonstrate dual improvements in interactive efficiency and accessibility across representative scientific computing scenarios.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While powerful and well-established, tools like ParaView present a steep learning curve that discourages many potential users. This work introduces ParaView-MCP, an autonomous agent that integrates modern multimodal large language models (MLLMs) with ParaView to not only lower the barrier to entry but also augment ParaView with intelligent decision support. By leveraging the state-of-the-art reasoning, command execution, and vision capabilities of MLLMs, ParaView-MCP enables users to interact with ParaView through natural language and visual inputs. Specifically, our system adopted the Model Context Protocol (MCP) - a standardized interface for model-application communication - that facilitates direct interaction between MLLMs with ParaView's Python API to allow seamless information exchange between the user, the language model, and the visualization tool itself. Furthermore, by implementing a visual feedback mechanism that allows the agent to observe the viewport, we unlock a range of new capabilities, including recreating visualizations from examples, closed-loop visualization parameter updates based on user-defined goals, and even cross-application collaboration involving multiple tools. Broadly, we believe such an agent-driven visualization paradigm can profoundly change the way we interact with visualization tools. We expect a significant uptake in the development of such visualization tools, in both visualization research and industry.

Problem

• Reduces ParaView's steep learning curve for users
• Enables natural language interaction with visualization tools
• Facilitates intelligent decision support via MLLMs

Innovation

• Integrates MLLMs with ParaView for intelligent support
• Uses Model Context Protocol for seamless API interaction
• Implements visual feedback for dynamic visualization updates