Athanor: Authoring Action Modification-based Interactions on Static Visualizations via Natural Language

2026-01-25

AI Summary

This work proposes a novel approach to automatically transform static visualizations into interactive ones through natural language instructions, eliminating the need for user programming and circumventing dependencies on original source code or data. The method leverages a multimodal large language model (MLLM) and introduces three key innovations: a structured “action-modification” interaction design space, a multi-agent intent parser that accurately interprets user requests, and a visualization abstraction transformer that maps semantic intent to concrete interactive behaviors. Through case studies and user interviews, the authors demonstrate that the proposed framework effectively supports diverse interaction scenarios while offering high flexibility and usability, thereby lowering the barrier for non-experts to create interactive visualizations from static figures.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Web data visualization

Abstract

Interactivity is crucial for effective data visualizations. However, it is often challenging to implement interactions for existing static visualizations, since the underlying code and data for existing static visualizations are often not available, and it also takes significant time and effort to enable interactions for them even if the original code and data are available. To fill this gap, we propose Athanor, a novel approach to transform existing static visualizations into interactive ones using multimodal large language models (MLLMs) and natural language instructions. Our approach introduces three key innovations: (1) an action-modification interaction design space that maps visualization interactions into user actions and corresponding adjustments, (2) a multi-agent requirement analyzer that translates natural language instructions into an actionable operational space, and (3) a visualization abstraction transformer that converts static visualizations into flexible and interactive representations regardless of their underlying implementation. Athanor allows users to effortlessly author interactions through natural language instructions, eliminating the need for programming. We conducted two case studies and in-depth interviews with target users to evaluate our approach. The results demonstrate the effectiveness and usability of our approach in allowing users to conveniently enable flexible interactions for static visualizations.

Problem

• static visualizations
• interaction authoring
• natural language
• data visualization
• interactivity

Innovation

• action-modification interaction
• multimodal large language models
• natural language authoring
• visualization abstraction
• interactive visualization