Shared Representation for 3D Pose Estimation, Action Classification, and Progress Prediction from Tactile Signals

2026-03-26

AI Summary

This work proposes SCOTTI, a shared convolutional Transformer model that unifies 3D human pose estimation, action classification, and action progress prediction through multi-task learning using tactile signals from wireless flexible insole sensors—a first in the field. Existing tactile approaches typically address these tasks independently, limiting performance, while vision-based methods suffer from occlusion and privacy concerns. Leveraging a newly collected dataset of over seven hours of foot pressure recordings from 15 participants performing eight daily activities, SCOTTI demonstrates significant improvements across all three tasks compared to conventional single-task baselines, thereby validating the efficacy and novelty of the shared representation framework.

Abstract

Estimating human pose, classifying actions, and predicting movement progress are essential for human-robot interaction. While vision-based methods suffer from occlusion and privacy concerns in realistic environments, tactile sensing avoids these issues. However, prior tactile-based approaches handle each task separately, leading to suboptimal performance. In this study, we propose a Shared COnvolutional Transformer for Tactile Inference (SCOTTI) that learns a shared representation to simultaneously address three separate prediction tasks: 3D human pose estimation, action class categorization, and action completion progress estimation. To the best of our knowledge, this is the first work to explore action progress prediction using foot tactile signals from custom wireless insole sensors. This unified approach leverages the mutual benefits of multi-task learning, enabling the model to achieve improved performance across all three tasks compared to learning them independently. Experimental results demonstrate that SCOTTI outperforms existing approaches across all three tasks. Additionally, we introduce a novel dataset collected from 15 participants performing various activities and exercises, with 7 hours of total duration, across eight different activities.

Problem

• 3D pose estimation
• action classification
• progress prediction
• tactile sensing
• multi-task learning

Innovation

• shared representation
• tactile sensing
• multi-task learning
• action progress prediction
• 3D pose estimation