XMTC: Explainable Early Classification of Multivariate Time Series in Reach-to-Grasp Hand Kinematics

2025-02-06

AI Summary

This study addresses the challenge of early and reliable recognition of human grasping intent in human-computer interaction (HCI). We propose the first framework integrating multi-model weighted ensemble learning with collaborative multi-view interpretability analysis. Methodologically, it leverages multivariate time-series kinematic hand-motion data from multiple sensors, jointly applying temporal classification, ensemble learning, and interpretability techniques—including temporal accuracy curves, confidence heatmaps, and partial dependence plots—to achieve high-accuracy prediction before 50% action completion. Key contributions include: (1) enabling early-prediction–confidence trade-off analysis; (2) facilitating attribution-based diagnosis under challenging conditions; and (3) identifying critical discriminative features. Quantitative evaluation on a real-world HCI dataset reveals distinguishable vs. confusable action pairs, sensor-dimension sensitivity, and robust discriminative features—substantially improving both timeliness and interpretability of early grasping-intent classification.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Multi-instance/Multi-view Learning
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Hand kinematics can be measured in Human-Computer Interaction (HCI) with the intention to predict the user's intention in a reach-to-grasp action. Using multiple hand sensors, multivariate time series data are being captured. Given a number of possible actions on a number of objects, the goal is to classify the multivariate time series data, where the class shall be predicted as early as possible. Many machine-learning methods have been developed for such classification tasks, where different approaches produce favorable solutions on different data sets. We, therefore, employ an ensemble approach that includes and weights different approaches. To provide a trustworthy classification production, we present the XMTC tool that incorporates coordinated multiple-view visualizations to analyze the predictions. Temporal accuracy plots, confusion matrix heatmaps, temporal confidence heatmaps, and partial dependence plots allow for the identification of the best trade-off between early prediction and prediction quality, the detection and analysis of challenging classification conditions, and the investigation of the prediction evolution in an overview and detail manner. We employ XMTC to real-world HCI data in multiple scenarios and show that good classification predictions can be achieved early on with our classifier as well as which conditions are easy to distinguish, which multivariate time series measurements impose challenges, and which features have most impact.

Problem

• Early classification of hand kinematics data
• Ensemble approach for multivariate time series
• Explainable predictions with multiple visualizations

Innovation

• Ensemble approach for classification
• XMTC tool with visualizations
• Early prediction with temporal accuracy