T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition

2026-06-19

AI Summary

Existing vision-language models struggle to explicitly model the temporal structure and bodily dynamics of human actions, limiting fine-grained, human-centric action understanding. To address this, this work proposes T-MOR, a novel framework that, for the first time, aligns skeletal sequences with vision-language representations through multimodal contrastive learning, enabling strong generalization in action recognition with only lightweight skeletal input. We introduce PoseCap-1M, a large-scale dataset comprising over one million video–text–pose triplets, and demonstrate significant improvements in both action classification and temporal action detection on benchmarks such as Toyota Smarthome and Penn Action. The model further exhibits remarkable generalization capabilities under few-shot and zero-shot settings.

Abstract

Vision-language models such as CLIP have recently achieved strong performance on a wide range of visual understanding tasks. However, most existing models rely primarily on appearance-level supervision from images or videos, and do not explicitly model human motion, which is essential for fine-grained and human-centric action recognition task as actions are defined by temporally structured and physically grounded body movements. To address this problem, we propose Transferable skeleton MOtion Representation (T-MOR), a motion-aware framework that learns transferable action representations from skeleton sequences with the aid of video and language supervision during training. T-MOR adopts a multi-modal contrastive learning scheme that aligns skeleton motion with visual and textual representations, while performing inference using only lightweight skeleton inputs. To support large-scale pre-training, we construct PoseCap-1M, a new dataset that contains over one million synchronized video, skeleton, and text triplets covering diverse human activities. We evaluate T-MOR on a range of human-centric action recognition benchmarks, including action classification and frame-wise temporal detection. Experimental results show that T-MOR consistently improves performance across multiple datasets, such as Toyota Smarthome, Penn Action, UAV-Human, TSU, and Charades. In addition, T-MOR demonstrates strong generalization ability in few-shot and zero-shot settings, highlighting the effectiveness of motion-centric and embodied representations for transferable action understanding.

Problem

• human action recognition
• motion modeling
• skeleton representation
• vision-language models
• fine-grained action understanding

Innovation

• skeleton-based action recognition
• motion-aware representation
• multimodal contrastive learning
• transferable representation
• PoseCap-1M