Prompt-guided Representation Disentanglement for Action Recognition

2025-09-25

AI Summary

In multi-action videos, conventional unified feature modeling leads to action interaction ambiguity, hindering precise action separation and recognition. Method: This paper proposes a prompt-guided representation disentanglement framework to achieve accurate isolation and identification of target actions. It introduces a Dynamic Prompt Module (DPM) to guide a graph parsing network, jointly modeling fine-grained action–object–relation structures via Spatio-Temporal Scene Graphs (SSGs). A video-adaptive Graph Parsing Neural Network (GPNN) and a dynamic weighting aggregation mechanism enable flexible, context-aware action representation disentanglement. Contribution/Results: The method supports action-directed extraction under arbitrary semantic prompts and achieves significant improvements over state-of-the-art methods on multiple multi-action benchmarks, demonstrating strong effectiveness and generalization capability in complex, cluttered scenes.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Prompt Engineering / Prompting
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Action recognition is a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects in multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as an effective solution. In this paper, we propose Prompt-guided Disentangled Representation for Action Recognition (ProDA), a novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design a video-adapted GPNN that aggregates information using dynamic weights. Experiments in video action recognition demonstrate the effectiveness of our approach when compared with the state-of-the-art methods. Our code can be found in https://github.com/iamsnaping/ProDA.git

Problem

• Disentangling specified actions from complex multi-action video scenes
• Modeling object interactions in multi-action scenarios using disentangled representations
• Generating action-specific representations through prompt-guided graph parsing networks

Innovation

• Prompt-guided disentangled representation for action recognition
• Spatio-temporal Scene Graphs with Dynamic Prompt Module
• Video-adapted Graph Parsing Neural Network with dynamic weights