SHARDeg: A Benchmark for Skeletal Human Action Recognition in Degraded Scenarios

2025-05-23

AI Summary

Existing skeletal human action recognition (SHAR) models lack robustness evaluation under realistic temporal degradations—such as frame-rate reduction, frame dropping, and jitter—commonly encountered on edge devices. Method: We introduce the first systematic benchmark for SHAR under temporal degradation, built upon NTU-RGB+D-120, quantitatively revealing that these three degradation types induce accuracy fluctuations exceeding 40%, with frame-timing regularity identified as the primary determinant of performance variance. We propose a time-series interpolation strategy to mitigate degradation effects, improving mainstream model accuracy by over 40%. Furthermore, we pioneer the application of Rough Path Theory to model temporal dynamics in SHAR, yielding LogSigRNN—a novel architecture that outperforms the SOTA DeGCN in 5 out of 6 low-frame-rate scenarios (average +6% accuracy), while exhibiting lower accuracy (−11–12%) on the original undegraded data, thereby validating the critical importance of degradation-aware modeling.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Learning on the Edge & Model Compression
• Planning, Routing, and Scheduling: Temporal Planning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Computer vision (CV) models for detection, prediction or classification tasks operate on video data-streams that are often degraded in the real world, due to deployment in real-time or on resource-constrained hardware. It is therefore critical that these models are robust to degraded data, but state of the art (SoTA) models are often insufficiently assessed with these real-world constraints in mind. This is exemplified by Skeletal Human Action Recognition (SHAR), which is critical in many CV pipelines operating in real-time and at the edge, but robustness to degraded data has previously only been shallowly and inconsistently assessed. Here we address this issue for SHAR by providing an important first data degradation benchmark on the most detailed and largest 3D open dataset, NTU-RGB+D-120, and assess the robustness of five leading SHAR models to three forms of degradation that represent real-world issues. We demonstrate the need for this benchmark by showing that the form of degradation, which has not previously been considered, has a large impact on model accuracy; at the same effective frame rate, model accuracy can vary by>40% depending on degradation type. We also identify that temporal regularity of frames in degraded SHAR data is likely a major driver of differences in model performance, and harness this to improve performance of existing models by up to>40%, through employing a simple mitigation approach based on interpolation. Finally, we highlight how our benchmark has helped identify an important degradation-resistant SHAR model based in Rough Path Theory; the LogSigRNN SHAR model outperforms the SoTA DeGCN model in five out of six cases at low frame rates by an average accuracy of 6%, despite trailing the SoTA model by 11-12% on un-degraded data at high frame rates (30 FPS).

Problem

• Assessing robustness of SHAR models to degraded video data
• Impact of degradation type on model accuracy variation
• Identifying degradation-resistant models for real-world SHAR applications

Innovation

• Benchmark for degraded SHAR on NTU-RGB+D-120
• Interpolation improves model performance by 40%
• LogSigRNN outperforms SoTA in low frame rates