Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study

2025-12-24

AI Summary

Language and voice abnormalities in neuropsychiatric disorders—Alzheimer’s disease (AD), depression, and autism spectrum disorder (ASD)—constitute promising early biomarkers; however, existing multimodal approaches suffer from poor cross-lingual generalizability and inconsistent evaluation protocols. To address these limitations, we propose FEND: a unified assessment framework spanning the full lifespan, integrating speech and text modalities, and supporting five languages (English, Chinese, Greek, French, Dutch). We introduce the first benchmark comprising 13 datasets across all three disorders and five languages. FEND incorporates large-model-based cross-lingual speech–text joint representation learning, standardized cross-dataset evaluation protocols, and a novel modality importance quantification method. Experiments show that multimodal fusion significantly outperforms unimodal baselines in AD and depression detection; ASD performance degradation is primarily attributable to dataset heterogeneity. Cross-lingual tasks achieve >86% accuracy under homogeneous conditions but drop by 12–19% in heterogeneous settings.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Neuropsychiatric disorders, such as Alzheimer's disease (AD), depression, and autism spectrum disorder (ASD), are characterized by linguistic and acoustic abnormalities, offering potential biomarkers for early detection. Despite the promise of multi-modal approaches, challenges like multi-lingual generalization and the absence of a unified evaluation framework persist. To address these gaps, we propose FEND (Foundation model-based Evaluation of Neuropsychiatric Disorders), a comprehensive multi-modal framework integrating speech and text modalities for detecting AD, depression, and ASD across the lifespan. Leveraging 13 multi-lingual datasets spanning English, Chinese, Greek, French, and Dutch, we systematically evaluate multi-modal fusion performance. Our results show that multi-modal fusion excels in AD and depression detection but underperforms in ASD due to dataset heterogeneity. We also identify modality imbalance as a prevalent issue, where multi-modal fusion fails to surpass the best mono-modal models. Cross-corpus experiments reveal robust performance in task- and language-consistent scenarios but noticeable degradation in multi-lingual and task-heterogeneous settings. By providing extensive benchmarks and a detailed analysis of performance-influencing factors, FEND advances the field of automated, lifespan-inclusive, and multi-lingual neuropsychiatric disorder assessment. We encourage researchers to adopt the FEND framework for fair comparisons and reproducible research.

Problem

• Develops a multi-modal framework for detecting neuropsychiatric disorders across lifespan
• Addresses challenges in multi-lingual generalization and unified evaluation for disorder assessment
• Evaluates performance of speech-text fusion across diverse languages and datasets

Innovation

• Multi-modal framework integrating speech and text
• Leveraging 13 multi-lingual datasets for evaluation
• Extensive benchmarks for automated lifespan-inclusive assessment