Explainable Parkinsons Disease Gait Recognition Using Multimodal RGB-D Fusion and Large Language Models

2025-12-03

AI Summary

To address the poor robustness and limited clinical interpretability of unimodal approaches in Parkinson’s disease (PD) gait recognition, this paper proposes an interpretable multimodal framework integrating RGB-D sensing and large language models (LLMs). Methodologically, we design a dual YOLOv11 encoder with a multi-scale local-global feature extraction module and introduce a cross-spatial fusion mechanism to achieve precise RGB-depth alignment. Crucially, we pioneer the use of a frozen pre-trained LLM as a vision-language mapper—transforming learned gait features into clinically meaningful textual explanations. Evaluated on a custom-built multimodal PD gait dataset, our method significantly enhances recognition stability under challenging conditions—including illumination variations and partial occlusions—achieving a +5.2% accuracy gain. Moreover, generated explanations align with clinical domain knowledge, providing actionable diagnostic rationale. The framework thus achieves a balanced triad: high accuracy, strong robustness, and transparent, human-readable decision support.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Accurate and interpretable gait analysis plays a crucial role in the early detection of Parkinsons disease (PD),yet most existing approaches remain limited by single-modality inputs, low robustness, and a lack of clinical transparency. This paper presents an explainable multimodal framework that integrates RGB and Depth (RGB-D) data to recognize Parkinsonian gait patterns under realistic conditions. The proposed system employs dual YOLOv11-based encoders for modality-specific feature extraction, followed by a Multi-Scale Local-Global Extraction (MLGE) module and a Cross-Spatial Neck Fusion mechanism to enhance spatial-temporal representation. This design captures both fine-grained limb motion (e.g., reduced arm swing) and overall gait dynamics (e.g., short stride or turning difficulty), even in challenging scenarios such as low lighting or occlusion caused by clothing. To ensure interpretability, a frozen Large Language Model (LLM) is incorporated to translate fused visual embeddings and structured metadata into clinically meaningful textual explanations. Experimental evaluations on multimodal gait datasets demonstrate that the proposed RGB-D fusion framework achieves higher recognition accuracy, improved robustness to environmental variations, and clear visual-linguistic reasoning compared with single-input baselines. By combining multimodal feature learning with language-based interpretability, this study bridges the gap between visual recognition and clinical understanding, offering a novel vision-language paradigm for reliable and explainable Parkinsons disease gait analysis. Code:https://github.com/manaralnaasan/RGB-D_parkinson-LLM

Problem

• Develops an explainable multimodal framework for Parkinson's gait recognition
• Integrates RGB-D data to improve accuracy and robustness in real conditions
• Uses LLMs to translate visual features into clinical explanations

Innovation

• Multimodal RGB-D fusion for Parkinsonian gait recognition
• Dual YOLOv11 encoders with cross-spatial fusion mechanism
• Frozen LLM translates visual features into clinical explanations