Severe Domain Shift in Skeleton-Based Action Recognition:A Study of Uncertainty Failure in Real-World Gym Environments

2026-03-16

AI Summary

This study addresses the performance degradation and safety risks in skeleton-based action recognition when transferring models from controlled multi-view 3D environments to real-world monocular 2D settings, where compound domain shifts induce severe model failure. The work reveals, for the first time, that high out-of-distribution (OOD) detection AUROC does not guarantee safety in selective classification. To evaluate zero-shot transfer, the authors introduce Gym2D and a real-world fitness dataset derived from UCF101, demonstrating catastrophic performance collapse (accuracy drops from 63.2% to 1.6%) and miscalibrated uncertainty. They propose a novel OOD detection signal combining energy scores and Mahalanobis distance, along with a lightweight fine-tuned gating mechanism for risk-aware rejection decisions. Experiments show the approach significantly reduces high-confidence erroneous predictions, thereby enhancing deployment safety in open-world scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Data transparency and provenance

Abstract

The practical deployment gap -- transitioning from controlled multi-view 3D skeleton capture to unconstrained monocular 2D pose estimation -- introduces a compound domain shift whose safety implications remain critically underexplored. We present a systematic study of this severe domain shift using a novel Gym2D dataset (style/viewpoint shift) and the UCF101 dataset (semantic shift). Our Skeleton Transformer achieves 63.2% cross-subject accuracy on NTU-120 but drops to 1.6% under zero-shot transfer to the Gym domain and 1.16% on UCF101. Critically, we demonstrate that high Out-Of-Distribution (OOD) detection AUROC does not guarantee safe selective classification. Standard uncertainty methods fail to detect this performance drop: the model remains confidently incorrect with 99.6% risk even at 50% coverage across both OOD datasets. While energy-based scoring (AUROC >= 0.91) and Mahalanobis distance provide reliable distributional detection signals, such high AUROC scores coexist with poor risk-coverage behavior when making decisions. A lightweight finetuned gating mechanism restores calibration and enables graceful abstention, substantially reducing the rate of confident wrong predictions. Our work challenges standard deployment assumptions, providing a principled safety analysis of both semantic and geometric skeleton recognition deployment.

Problem

• domain shift
• skeleton-based action recognition
• uncertainty failure
• out-of-distribution detection
• real-world deployment

Innovation

• domain shift
• uncertainty calibration
• skeleton-based action recognition
• out-of-distribution detection
• selective classification