HiLWS: A Human-in-the-Loop Weak Supervision Framework for Curating Clinical and Home Video Data for Neurological Assessment

2025-09-09

AI Summary

To address challenges in Parkinson’s disease home-video assessment—including poor visual quality, inconsistent task execution, high annotation noise, and domain shift—this paper proposes a cascaded human-in-the-loop weakly supervised framework. The method integrates pose estimation refinement, video-quality-aware adaptive filtering, precise task-segment extraction, and context-sensitive prioritization of ambiguous samples to establish an end-to-end video data cleaning pipeline. It further employs a two-stage weakly supervised modeling strategy coupled with dynamic expert feedback to iteratively improve label quality. Key innovations include a context-aware clinical evaluation metric to guide manual review and a robust transfer paradigm bridging clinical and home-video domains. Experiments demonstrate significant improvements in label consistency (+28.6%) and model generalizability, validating the framework’s effectiveness and clinical applicability in cross-scenario neuromotor assessment.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Web data integration and cleaning

Abstract

Video-based assessment of motor symptoms in conditions such as Parkinson's disease (PD) offers a scalable alternative to in-clinic evaluations, but home-recorded videos introduce significant challenges, including visual degradation, inconsistent task execution, annotation noise, and domain shifts. We present HiLWS, a cascaded human-in-the-loop weak supervision framework for curating and annotating hand motor task videos from both clinical and home settings. Unlike conventional single-stage weak supervision methods, HiLWS employs a novel cascaded approach, first applies weak supervision to aggregate expert-provided annotations into probabilistic labels, which are then used to train machine learning models. Model predictions, combined with expert input, are subsequently refined through a second stage of weak supervision. The complete pipeline includes quality filtering, optimized pose estimation, and task-specific segment extraction, complemented by context-sensitive evaluation metrics that assess both visual fidelity and clinical relevance by prioritizing ambiguous cases for expert review. Our findings reveal key failure modes in home recorded data and emphasize the importance of context-sensitive curation strategies for robust medical video analysis.

Problem

• Addressing annotation noise and domain shifts in home-recorded neurological videos
• Developing cascaded human-in-the-loop weak supervision for clinical video curation
• Improving motor symptom assessment reliability in Parkinson's disease videos

Innovation

• Cascaded human-in-the-loop weak supervision
• Quality filtering and optimized pose estimation
• Context-sensitive evaluation metrics prioritizing ambiguous cases