InvestChat: Exploring Multimodal Interaction via Natural Language, Touch, and Pen in an Investment Dashboard

2026-04-21

AI Summary

This study addresses the low engagement of novice investors in stock market exploration by designing and implementing a multimodal investment dashboard that integrates natural language, touch, and stylus inputs. The system leverages a large language model–powered conversational interface alongside coordinated multi-view visualizations, enabling users on tablet devices to fluidly switch among interaction modalities. This work represents the first integration of these three input modalities within an investment analysis context, demonstrating their complementary nature and effectiveness. User studies reveal that the multimodal design significantly enhances novice users’ sense of engagement, with natural language interaction being the most preferred modality, and that the combined use of all modalities yields superior performance compared to any single mode alone.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: The sharing economy
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

We designed and implemented InvestChat, a multimodal tablet-based application that supports stock market exploration with multiple coordinated views and an LLM-powered chat. We evaluated the application with 12 novice investors. Our findings suggest that combining natural language, touch, and pen input during stock market exploration facilitates user engagement. Participants leveraged the modalities in complementary ways, enjoying the freedom of choice and finding natural language most effective.

Problem

• multimodal interaction
• investment dashboard
• stock market exploration
• natural language interface
• novice investors

Innovation

• multimodal interaction
• natural language interface
• LLM-powered chat
• investment dashboard
• pen and touch input