Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training

2026-06-26

AI Summary

This work addresses the limitations of single-view skeletal approaches in assessing hand motion quality during Traditional Chinese Medicine (TCM) rehabilitation training, where severe self-occlusion and complex hand-object interactions commonly occur. To overcome these challenges, we propose the first cross-view multimodal framework that integrates first-person and third-person video streams for action quality assessment. By jointly modeling dual perspectives and fusing visual and pose features, our method enhances contextual awareness of the environment and effectively mitigates occlusion issues. We contribute two expert-annotated dual-view TCM rehabilitation datasets, TCM-AQA61-A and TCM-AQA61-T, and demonstrate significant performance gains over existing methods: weighted F1 scores improve by over 10% on needle insertion depth and rapid insertion tasks, while mean absolute errors for insertion duration and operation frequency are notably reduced.

Abstract

Vision-based assessment can provide convenient and cost-effective evaluation in Traditional Chinese Medicine (TCM) rehabilitation training, where action quality assessment (AQA) from computer vision offers a promising solution. Existing automatic AQA frameworks for physical therapy typically rely on skeletal data captured from a single viewpoint, which is inefficient for TCM techniques such as acupuncture or Tuina that involve dense hand self-occlusion and complex hand-object interactions. To address these challenges, we propose CME-AQA, a cross-view, multimodal vision-based assessment framework that integrates visual-pose fusion to enhance understanding of environmental context and leverages both first-person and third-person videos during training to improve inference robustness. We collected two dual-view datasets, TCM-AQA61-A (Acupuncture) and TCM-AQA61-T (Tuina), each containing synchronized first-person and third-person recordings of 61 subjects with expert annotations. Experimental results show that our approach achieves superior or comparable mean performance against competitive baselines, achieving over 10% relative improvement in weighted F1 over the best competing method on key rating tasks such as Needle Depth and Quick Needle Insertion, while also reducing mean absolute error in quantitative measures such as insertion time and manipulation frequency. Testing on a CPR dataset further demonstrates comparable performance on several posture-based criteria, suggesting applicability to related structured simulated clinical skill assessments where participant motion is central to evaluation. Overall, CME-AQA enhances assessment accuracy for structured TCM rehabilitation training and facilitates more convenient and effective training-oriented skill evaluation.

Problem

• Action Quality Assessment
• Traditional Chinese Medicine
• Hand Self-Occlusion
• Multimodal Vision
• Rehabilitation Training

Innovation

• cross-view
• multimodal fusion
• action quality assessment
• first-person vision
• TCM rehabilitation