Developing Conversational Speech Systems for Robots to Detect Speech Biomarkers of Cognition in People Living with Dementia

2025-02-15

AI Summary

Real-time assessment of speech-based biomarkers for early dementia screening remains challenging in clinical practice. Method: This study introduces the first clinically oriented conversational speech analysis robot system, integrating a fine-tuned domain-specific large language model (dementia-LLM) with a low-latency WebSocket architecture. The frontend employs a Progressive Web App (PWA), while the backend combines speech feature engineering with six-dimensional cognitive-relevant biomarker modeling, culminating in a Composite Voice-Based Score (CVBS). Contribution/Results: The system achieves end-to-end, real-time (response latency <1.5 s), and interpretable quantification of cognitive status—the first of its kind. On the DementiaBank corpus, CVBS demonstrates moderate correlation with the Mini-Mental State Examination (MMSE; r = 0.58), significantly outperforming individual speech biomarkers. Additionally, the system supports automated, clinician-friendly visualization reports, underscoring its readiness for clinical deployment.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Conversational AI/Dialog Systems
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This study presents the development and testing of a conversational speech system designed for robots to detect speech biomarkers indicative of cognitive impairments in people living with dementia (PLwD). The system integrates a backend Python WebSocket server and a central core module with a large language model (LLM) fine-tuned for dementia to process user input and generate robotic conversation responses in real-time in less than 1.5 seconds. The frontend user interface, a Progressive Web App (PWA), displays information and biomarker score graphs on a smartphone in real-time to human users (PLwD, caregivers, clinicians). Six speech biomarkers based on the existing literature - Altered Grammar, Pragmatic Impairments, Anomia, Disrupted Turn-Taking, Slurred Pronunciation, and Prosody Changes - were developed for the robot conversation system using two datasets, one that included conversations of PLwD with a human clinician (DementiaBank dataset) and one that included conversations of PLwD with a robot (Indiana dataset). We also created a composite speech biomarker that combined all six individual biomarkers into a single score. The speech system's performance was first evaluated on the DementiaBank dataset showing moderate correlation with MMSE scores, with the composite biomarker score outperforming individual biomarkers. Analysis of the Indiana dataset revealed higher and more variable biomarker scores, suggesting potential differences due to study populations (e.g. severity of dementia) and the conversational scenario (human-robot conversations are different from human-human). The findings underscore the need for further research on the impact of conversational scenarios on speech biomarkers and the potential clinical applications of robotic speech systems.

Problem

• Detect cognitive impairments in dementia
• Develop conversational speech system for robots
• Evaluate speech biomarkers in human-robot interactions

Innovation

• Python WebSocket server integration
• LLM fine-tuned for dementia
• Real-time PWA biomarker display