Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening

2026-06-10

AI Summary

This study addresses the challenges of heterogeneous data fusion and imprecise disease staging in voice-based screening for neurodegenerative disorders such as Alzheimer’s and Parkinson’s diseases. To this end, the authors propose NeurMLLM, a novel framework that leverages multimodal large language models for fine-grained disease staging. The approach employs Vision Transformers to encode spectrogram and MFCC features from speech, maps them into the embedding space of a large language model, and concatenates them with textual transcripts and demographic information into a unified sequence. The model is then instruction-tuned end-to-end using LoRA for generative classification. Evaluated on the Bridge2AI-Voice dataset, NeurMLLM significantly outperforms conventional machine learning and existing LLM-based methods, achieving markedly higher staging accuracy and demonstrating strong potential for clinical deployment.

Abstract

Voice-based screening offers a scalable and non-invasive way to assess neurodegenerative diseases such as Alzheimer's disease (AD) and Parkinson's disease (PD), but their staging remains challenging due to the difficulty of integrating heterogeneous data. This paper presents NeurMLLM, an efficient multimodal generative framework for neurodegenerative disease staging. NeurMLLM first encodes the spectrograms and Mel-frequency cepstral coefficients of audio data with vision transformers and projects their representations into the embedding space of a large language model (LLM), where they are concatenated with transcript and demographic instruction tokens as a single unified sequence. The LLM is then instruction-tuned via Low-Rank Adaptation using task prompts to autoregressively predict a constrained label token, enabling a generative classification. By evaluating on the Bridge2AI-Voice dataset for fine-grained staging of AD and PD, we observe that NeurMLLM achieves strong performance, consistently outperforming classical machine learning methods and existing LLM-based approaches. The results show the high potential of multimodal LLMs in neurodegenerative disease staging, improving staging accuracy and supporting accessible deployment.

Problem

• neurodegenerative disease staging
• multimodal integration
• voice-based screening
• heterogeneous data

Innovation

• Multimodal LLMs
• Neurodegenerative disease staging
• Acoustic-text fusion
• Low-Rank Adaptation
• Generative classification