SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos

2026-06-30

AI Summary

This work addresses the challenge of accurately assessing the quality of users’ skilled movements in real-time personalized instruction scenarios. To this end, we propose SkillSpotter, a pose-aware multi-view architecture that jointly performs action detection and quality scoring through adaptive temporal suppression, gated 3D human pose fusion, and bidirectional cross-view attention. Our approach is the first to integrate human kinematic cues with visual features for action quality assessment and introduces a novel evaluation protocol that decouples detection from scoring, along with a generalizable and transferable module design. Evaluated on Ego-Exo4D, SkillSpotter achieves a 76% relative improvement in class-specific mAP (from 12.40 to 21.82) and a balanced accuracy of 60.40%, while demonstrating strong generalization on HoloAssist.

Abstract

To enable personalized, real-time coaching using Augmented Reality glasses or fixed camera setups in domains such as sports, cooking, or music, a system must understand not just what a person does, but how well they execute an activity. In an ego-exo video setting, this requires simultaneously detecting individual skilled actions and classifying each as correct or needing improvement, which Ego-Exo4D's proficiency demonstration benchmark formalized. We first adapt seven state-of-the-art temporal action detection architectures to this task, extend the evaluation protocol to disentangle detection from grading, and show that existing methods grade near-randomly. We then introduce SkillSpotter, a pose-aware multi-view architecture that jointly detects and grades skilled actions through three task-specific modules: (1) adaptive temporal suppression to handle the varying density of skilled actions across diverse activities, (2) gated 3D body pose fusion to leverage body kinematics as a complementary signal to visual features, and (3) bidirectional cross-view attention to combine ego and exo views effectively. SkillSpotter improves class-specific mAP from 12.40 to 21.82 (+76%) and balanced accuracy from 55.99% to 60.40% over the best baseline. SkillSpotter's modules transfer to other temporal action detection models with consistent gains, and our method generalizes beyond Ego-Exo4D to HoloAssist. Code: https://github.com/eth-siplab/SkillSpotter

Problem

• skilled action detection
• action grading
• ego-exo videos
• proficiency assessment
• temporal action detection

Innovation

• pose-aware
• multi-view
• temporal action detection
• action grading
• cross-view attention