Real-time Speech Summarization for Medical Conversations

2024-06-22

Interspeech

AI Summary

Real-time speech summarization for doctor–patient dialogues remains underexplored, particularly for clinical deployment. Method: We propose the first deployable real-time medical dialogue speech summarization system, capable of generating turn-level local summaries during interaction and a final global summary upon dialogue completion. Our approach integrates automatic speech recognition (ASR), sequential abstractive summarization, and large language models (LLMs) within a novel real-time segmentation architecture. To address data scarcity, we introduce VietMed-Sum—a high-quality, bilingual (English–Vietnamese) medical speech summarization dataset—curated via a human-in-the-loop annotation paradigm combining LLM-based pre-screening and expert refinement. Contribution/Results: Our method achieves state-of-the-art performance on VietMed-Sum. All code, the dataset, and pretrained models are publicly released to foster reproducible research and clinical adoption.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In doctor-patient conversations, identifying medically relevant information is crucial, posing the need for conversation summarization. In this work, we propose the first deployable real-time speech summarization system for real-world applications in industry, which generates a local summary after every N speech utterances within a conversation and a global summary after the end of a conversation. Our system could enhance user experience from a business standpoint, while also reducing computational costs from a technical perspective. Secondly, we present VietMed-Sum which, to our knowledge, is the first speech summarization dataset for medical conversations. Thirdly, we are the first to utilize LLM and human annotators collaboratively to create gold standard and synthetic summaries for medical conversation summarization. Finally, we present baseline results of state-of-the-art models on VietMed-Sum. All code, data (English-translated and Vietnamese) and models are available online: https://github.com/leduckhai/MultiMed/tree/master/VietMed-Sum

Problem

• Real-time summarization of medical conversations
• First deployable system for industry applications
• Creating dataset using LLM and human collaboration

Innovation

• Real-time speech summarization system deployable
• First medical speech summarization dataset VietMed-Sum
• LLM and human collaboration for summaries