CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition

2024-10-09

arXiv.org

AI Summary

Multi-entity action recognition suffers from degraded performance of generic backbone networks due to large inter-entity variations in skeletal spatial distributions. To address this, we propose a sample-level convex-hull adaptive normalization method: an implicit convex-hull-constrained origin-shift mechanism jointly optimizes lightweight convex-combination coefficient learning, enabling geometry-aware skeletal repositioning. Additionally, we introduce mini-batch pairwise Maximum Mean Discrepancy (MMD) as a distribution alignment auxiliary objective to guide end-to-end optimization. Our approach requires no additional annotations and significantly improves the recognition accuracy of single-entity backbone models on multi-entity scenarios across six benchmarks—NTU Mutual, H2O, Assembly101, NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD. Code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Skeleton-based multi-entity action recognition is a challenging task aiming to identify interactive actions or group activities involving multiple diverse entities. Existing models for individuals often fall short in this task due to the inherent distribution discrepancies among entity skeletons, leading to suboptimal backbone optimization. To this end, we introduce a Convex Hull Adaptive Shift based multi-Entity action recognition method (CHASE), which mitigates inter-entity distribution gaps and unbiases subsequent backbones. Specifically, CHASE comprises a learnable parameterized network and an auxiliary objective. The parameterized network achieves plausible, sample-adaptive repositioning of skeleton sequences through two key components. First, the Implicit Convex Hull Constrained Adaptive Shift ensures that the new origin of the coordinate system is within the skeleton convex hull. Second, the Coefficient Learning Block provides a lightweight parameterization of the mapping from skeleton sequences to their specific coefficients in convex combinations. Moreover, to guide the optimization of this network for discrepancy minimization, we propose the Mini-batch Pair-wise Maximum Mean Discrepancy as the additional objective. CHASE operates as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies, thereby reducing data bias and improving the subsequent classifier's multi-entity action recognition performance. Extensive experiments on six datasets, including NTU Mutual 11/26, H2O, Assembly101, Collective Activity and Volleyball, consistently verify our approach by seamlessly adapting to single-entity backbones and boosting their performance in multi-entity scenarios. Our code is publicly available at https://github.com/Necolizer/CHASE .

Problem

• Multi-entity action recognition
• Object skeleton variation
• Position distribution discrepancy

Innovation

• CHASE
• Convex Hull Adaptive Shift Enhancement
• Multi-entity Action Recognition