Improving skeleton-based action recognition with interactive object information

2025-01-07

International Journal of Multimedia Information Retrieval

AI Summary

To address the lack of object-interaction semantic modeling in skeleton-based action recognition, this paper proposes an object-aware end-to-end action understanding framework. The core method introduces learnable and optimizable object representations to explicitly model spatio-temporal associations between human joints and scene objects. It incorporates an attention-driven interaction modeling mechanism, a multimodal feature fusion module, and a differentiable object-relation reasoning module—marking the first integration of explicit object information into graph convolutional network (GCN)-based skeleton action recognition architectures. Evaluated on NTU-60 and NTU-120 benchmarks, the approach achieves absolute accuracy improvements of 2.3% and 1.9%, respectively, significantly outperforming skeleton-only baselines. These results empirically validate the critical contribution of object-interaction semantics to action recognition performance.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Video Understanding & Activity Analysis
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Problem

• Action Recognition
• Skeleton-Based
• Object-Related Actions

Innovation

• ST-VGCN Network
• Item Node Integration
• Enhanced Action Recognition