From Retrieving Information to Reasoning with AI: Exploring Different Interaction Modalities to Support Human-AI Coordination in Clinical Decision-Making

2026-01-29

AI Summary

This study investigates how clinicians interact with large language models (LLMs) for clinical decision support, revealing that their use is predominantly limited to information retrieval rather than deep reasoning, and that they lack a nuanced understanding of how interaction modalities shape human-AI collaboration. Through qualitative analysis of multimodal interactions—including text, speech, and static or dynamic user interfaces—with 12 clinicians, the research demonstrates that practitioners primarily employ LLMs for verifying known information rather than engaging in complex diagnostic reasoning. Interaction modality significantly influences the depth of collaboration, and clinicians exhibit marked differences in modality preferences based on individual cognitive styles. These findings underscore the absence of a one-size-fits-all interaction paradigm and highlight the necessity of tailoring interface designs to users’ cognitive characteristics to enhance the efficacy of clinical AI collaboration.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Intelligent User Interfaces
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

LLMs are popular among clinicians for decision-support because of simple text-based interaction. However, their impact on clinicians'performance is ambiguous. Not knowing how clinicians use this new technology and how they compare it to traditional clinical decision-support systems (CDSS) restricts designing novel mechanisms that overcome existing tool limitations and enhance performance and experience. This qualitative study examines how clinicians (n=12) perceive different interaction modalities (text-based conversation with LLMs, interactive and static UI, and voice) for decision-support. In open-ended use of LLM-based tools, our participants took a tool-centric approach using them for information retrieval and confirmation with simple prompts instead of use as active deliberation partners that can handle complex questions. Critical engagement emerged with changes to the interaction setup. Engagement also differed with individual cognitive styles. Lastly, benefits and drawbacks of interaction with text, voice and traditional UIs for clinical decision-support show the lack of a one-size-fits-all interaction modality.

Problem

• clinical decision-making
• human-AI coordination
• interaction modalities
• large language models
• clinical decision-support systems

Innovation

• human-AI coordination
• interaction modalities
• clinical decision-support
• large language models
• cognitive styles