Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

2026-02-20

AI Summary

This work addresses the challenge in few-shot action recognition that coarse-grained class names alone are insufficient to capture the fine-grained spatiotemporal characteristics of novel actions. To overcome this limitation, the authors propose DiST, a novel framework that leverages large language models to explicitly disentangle action semantics into spatial and temporal commonsense descriptions. The framework introduces dedicated Spatial and Temporal Knowledge Compensators (SKC/TKC) to learn object-level and frame-level prototypes, respectively, thereby establishing an interpretable, multi-granularity prototypical learning mechanism. This approach significantly enhances modeling of fine-grained spatiotemporal patterns and achieves state-of-the-art performance across five standard benchmarks, outperforming existing methods by a notable margin.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multimodal Learning
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Few-Shot Action Recognition (FSAR) is a challenging task that requires recognizing novel action categories with a few labeled videos. Recent works typically apply semantically coarse category names as auxiliary contexts to guide the learning of discriminative visual features. However, such context provided by the action names is too limited to provide sufficient background knowledge for capturing novel spatial and temporal concepts in actions. In this paper, we propose DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes. In the decomposition stage, we decouple vanilla action names into diverse spatio-temporal attribute descriptions (action-related knowledge). Such commonsense knowledge complements semantic contexts from spatial and temporal perspectives. In the incorporation stage, we propose Spatial/Temporal Knowledge Compensators (SKC/TKC) to discover discriminative object-level and frame-level prototypes, respectively. In SKC, object-level prototypes adaptively aggregate important patch tokens under the guidance of spatial knowledge. Moreover, in TKC, frame-level prototypes utilize temporal attributes to assist in inter-frame temporal relation modeling. These learned prototypes thus provide transparency in capturing fine-grained spatial details and diverse temporal patterns. Experimental results show DiST achieves state-of-the-art results on five standard FSAR datasets.

Problem

• Few-Shot Action Recognition
• Spatio-temporal Knowledge
• Action Recognition
• Background Knowledge
• Spatial and Temporal Concepts

Innovation

• Spatio-temporal Decoupling
• Knowledge Compensation
• Few-Shot Action Recognition
• Prototype Learning
• Large Language Models