It Hears, It Sees too: Multi-Modal LLM for Depression Detection By Integrating Visual Understanding into Audio Language Models

2025-11-24

AI Summary

Existing depression detection methods predominantly rely on unimodal large language models operating solely on text, failing to effectively model critical nonverbal cues such as speech prosody and facial expressions; meanwhile, general-purpose multimodal large models lack domain-specific architectural design for psychological assessment. This paper proposes a dedicated multimodal large model for depression detection: built upon an audio-language foundation model, it integrates visual understanding capabilities and introduces a fine-grained, timestamp-level audiovisual feature alignment framework to enhance cross-modal temporal dynamic modeling. A novel cross-modal attention mechanism enables joint representation learning of nonverbal behavioral cues. The approach significantly reduces training resource requirements and supports seamless integration of physiological signals in future extensions. Evaluated on the DAIC-WoZ dataset, it outperforms state-of-the-art unimodal and multimodal baselines, demonstrating strong clinical applicability and scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Depression is one of the most prevalent mental health disorders globally. In recent years, multi-modal data, such as speech, video, and transcripts, has been increasingly used to develop AI-assisted depression assessment systems. Large language models have further advanced this field due to their strong language understanding and generalization capabilities. However, conventional LLMs remain text-centric and cannot process the rich non-verbal cues found in audio and visual modalities, which are critical components in mental health evaluation. While multi-modal LLMs offer a promising direction, few are tailored for psychological applications. In this study, we propose a novel multi-modal LLM framework for depression detection. Our approach augments an audio language model with visual understanding and aligns audio-visual features at the timestamp level. This fine-grained alignment improves modeling of temporal dynamics across modalities while reducing the need for extensive training data and computational resources. Experiments on the DAIC-WoZ dataset demonstrate that our model outperforms both single-modality approaches and previous multi-modal methods. Moreover, the proposed framework can be extended to incorporate additional physiological signals, paving the way for broader clinical applications beyond mental health.

Problem

• Develops multi-modal LLM integrating audio and visual cues for depression detection
• Addresses limitations of text-centric models in processing non-verbal behavioral signals
• Enables fine-grained temporal alignment of audio-visual features with reduced resource requirements

Innovation

• Integrates visual understanding into audio language models
• Aligns audio-visual features at timestamp level granularity
• Reduces need for extensive training data resources