SpeechDx: A Multi-Task Benchmark for Clinical Speech AI

2026-06-15

AI Summary

Clinical speech AI research has long been hindered by isolated disease-specific models and the absence of a unified evaluation benchmark, leading to incomparable results and difficulty in assessing generalization. This work proposes SpeechDx—the first large-scale multitask benchmark for clinical speech AI—integrating 12 datasets and 27 tasks, systematically organized according to stages of speech production (conceptualization, formulation, and articulation) to enable cross-disease zero-shot transfer and few-shot generalization evaluation. Through systematic evaluation of 12 state-of-the-art audio encoders, we find that large-scale speech foundation models consistently outperform others overall, whereas domain-specific models are effective only on highly matched tasks. Critically, no existing model demonstrates reliable generalization across the entire clinical speech domain.

Abstract

Speech offers a uniquely informative window into health by simultaneously engaging neurological, motor, respiratory, and vocal systems. Current clinical speech AI methods have largely progressed through isolated condition-specific studies, making results difficult to compare and generalization difficult to assess. We introduce SpeechDx, a large-scale benchmark for clinical speech AI spanning 12 datasets and 27 tasks across diverse health conditions. To enable evaluation across shared clinical mechanisms, SpeechDx structures tasks by the stage of speech production they disrupt: conceptualization, formulation, and articulation. The benchmark tests generalization by including tasks with limited labeled data and evaluating the same health condition across multiple datasets, distinguishing clinically meaningful patterns from dataset artefacts. We systematically evaluate 12 state-of-the-art audio encoders across all tasks and under zero-shot cross-condition transfer. Results show that large-scale speech models represent the strongest overall baselines, domain-specific models improve performance only on closely matched tasks, and no current representation generalizes reliably across the clinical speech landscape. SpeechDx establishes a shared evaluation framework for tracking progress toward general-purpose clinical speech representations

Problem

• clinical speech AI
• generalization
• benchmark
• multi-task
• speech production

Innovation

• clinical speech AI
• multi-task benchmark
• speech production stages
• zero-shot transfer
• generalization evaluation