Multimodal Biomarkers for Schizophrenia: Towards Individual Symptom Severity Estimation

2025-05-21

AI Summary

Current schizophrenia assessment predominantly relies on binary classification paradigms, overlooking symptom continuity and clinical heterogeneity—limiting real-world clinical utility. To address this, we propose the first multimodal deep learning framework explicitly designed for fine-grained, continuous symptom severity estimation. Our approach integrates speech (Wav2Vec 2.0), facial behavior (3D-CNN + LSTM), and textual (BERT) features, and introduces a gated cross-modal collaborative learning mechanism to enable interpretable, phenotype-aware modeling. Crucially, it shifts from conventional “ill/healthy” categorical diagnosis toward dimensional, continuous symptom quantification. Evaluated on a public multimodal mental health dataset, our method achieves a 23.6% average reduction in mean absolute error (MAE) across symptom dimensions compared to unimodal baselines and state-of-the-art classification models. The framework demonstrates strong potential for clinical deployment due to its robustness, interpretability, and alignment with dimensional psychiatry principles.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Studies on schizophrenia assessments using deep learning typically treat it as a classification task to detect the presence or absence of the disorder, oversimplifying the condition and reducing its clinical applicability. This traditional approach overlooks the complexity of schizophrenia, limiting its practical value in healthcare settings. This study shifts the focus to individual symptom severity estimation using a multimodal approach that integrates speech, video, and text inputs. We develop unimodal models for each modality and a multimodal framework to improve accuracy and robustness. By capturing a more detailed symptom profile, this approach can help in enhancing diagnostic precision and support personalized treatment, offering a scalable and objective tool for mental health assessment.

Problem

• Estimates individual schizophrenia symptom severity using multimodal data
• Overcomes oversimplification in traditional binary classification approaches
• Integrates speech, video, and text for precise diagnostic tools

Innovation

• Multimodal approach integrating speech, video, text
• Unimodal models combined for enhanced accuracy
• Detailed symptom estimation for personalized treatment