Partial Skeleton Visibility for Action Recognition: A Constrained Field-of-View Approach

2026-07-01

AI Summary

This work addresses the significant performance degradation of existing skeleton-based action recognition methods under limited fields of view, where missing joints impair model efficacy. To systematically tackle this challenge, the authors propose PartialVisGraph, a novel framework that introduces learnable virtual hyperedges to construct a hypergraph for modeling high-order joint dependencies. It further incorporates a single-head sample-adaptive Transformer fused with visibility priors and a visibility-gated mechanism to suppress the propagation of unreliable information from occluded joints. Evaluated on simulated limited-view benchmarks of NTU RGB+D 60/120, the method achieves state-of-the-art performance, improving accuracy by up to 68.8% on severely occluded subsets while maintaining superior results even with complete skeletons.

Abstract

Skeleton-based action recognition has achieved remarkable success by exploiting joint coordinates and their topological connections, yet prevailing methods overwhelmingly assume complete and clean skeleton inputs. In real-world deployments, such as egocentric vision, crowded surveillance, wearable devices, or edge robotics, limited field-of-view (FoV) frequently causes substantial joint visibility dropout, leading to severe performance degradation that existing models are largely unprepared to handle. To bridge this critical yet underexplored gap, we introduce PartialVisGraph, a novel hypergraph framework tailored for robust skeleton action recognition under constrained FoV. We first construct highly expressive hypergraphs by introducing learnable virtual hyperedges that form a soft incidence matrix, capturing flexible high-order dependencies beyond conventional pairwise graphs. We then propose the Single-Head Sample-Adaptive Transformer, which adaptively aggregates joint features onto hyperedges while explicitly incorporating a visibility prior. This prior selectively gates information flow, preventing occluded or out-of-view joints from corrupting reliable feature propagation. We further establish rigorous evaluation protocols with realistic FoV simulation benchmarks on NTU RGB+D 60 and 120. Extensive experiments demonstrate that PartialVisGraph consistently achieves state-of-the-art accuracy under partial visibility, with gains of up to 68.8\% on subsets with severe FoV restrictions compared to recent strong baselines, while remaining superior on full-visibility settings. Our approach offers a principled and practical pathway toward deployable skeleton-based action understanding in unconstrained environments.

Problem

• skeleton-based action recognition
• partial visibility
• field-of-view
• joint occlusion
• real-world deployment

Innovation

• partial skeleton visibility
• hypergraph learning
• visibility-aware transformer
• constrained field-of-view
• skeleton-based action recognition