MIND: Empowering Mental Health Clinicians with Multimodal Data Insights through a Narrative Dashboard

2026-01-21

AI Summary

This study addresses the challenge of effectively integrating patient-generated multimodal data—such as from wearable devices and self-report questionnaires—with clinical records in psychiatric practice. To bridge this gap, the authors collaborated with clinicians to design a narrative-driven dashboard that, for the first time, leverages large language models to synthesize heterogeneous data sources into coherent, context-aware natural language narratives grounded in clinical semantics, complemented by interactive visualizations. This approach substantially enhances the interpretability and clinical utility of complex data. In a user study involving 16 psychiatrists, the system demonstrated statistically significant improvements over baseline methods in revealing clinically relevant insights (p<.001) and supporting clinical decision-making (p=.004).

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Advances in data collection enable the capture of rich patient-generated data: from passive sensing (e.g., wearables and smartphones) to active self-reports (e.g., cross-sectional surveys and ecological momentary assessments). Although prior research has demonstrated the utility of patient-generated data in mental healthcare, significant challenges remain in effectively presenting these data streams along with clinical data (e.g., clinical notes) for clinical decision-making. Through co-design sessions with five clinicians, we propose MIND, a large language model-powered dashboard designed to present clinically relevant multimodal data insights for mental healthcare. MIND presents multimodal insights through narrative text, complemented by charts communicating underlying data. Our user study (N=16) demonstrates that clinicians perceive MIND as a significant improvement over baseline methods, reporting improved performance to reveal hidden and clinically relevant data insights (p<.001) and support their decision-making (p=.004). Grounded in the study results, we discuss future research opportunities to integrate data narratives in broader clinical practices.

Problem

• multimodal data
• mental health
• clinical decision-making
• data visualization
• patient-generated data

Innovation

• multimodal data integration
• large language model
• narrative dashboard
• clinical decision support
• mental health informatics