Adaptive Prototype Model for Attribute-based Multi-label Few-shot Action Recognition

2025-02-18

AI Summary

This work addresses few-shot action recognition under realistic multi-attribute (e.g., action, scene, object) joint annotation—a challenging setting where limited labeled samples and label bias hinder generalization. We propose the Adaptive Attribute Prototype Model (AAPM), which introduces a novel Textual Constraint Module (TCM) to guide attribute-specific prototype construction via semantic textual priors, and an Attribute Allocation Method (AAM) to mitigate label bias in multi-label settings. To support this research, we introduce Multi-Kinetics—the first attribute-driven, multi-label few-shot video dataset. AAPM unifies prototype learning, multi-label classification, and meta-learning through text-guided semantic constraints, cross-modal alignment, and attribute-decoupled representation modeling. Extensive experiments demonstrate state-of-the-art performance on both single-label and multi-label few-shot action recognition benchmarks grounded in multi-attribute bases, with significant gains in generalization and robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Multiagent Systems: Multiagent Learning

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In real-world action recognition systems, incorporating more attributes helps achieve a more comprehensive understanding of human behavior. However, using a single model to simultaneously recognize multiple attributes can lead to a decrease in accuracy. In this work, we propose a novel method i.e. Adaptive Attribute Prototype Model (AAPM) for human action recognition, which captures rich action-relevant attribute information and strikes a balance between accuracy and robustness. Firstly, we introduce the Text-Constrain Module (TCM) to incorporate textual information from potential labels, and constrain the construction of different attributes prototype representations. In addition, we explore the Attribute Assignment Method (AAM) to address the issue of training bias and increase robustness during the training process.Furthermore, we construct a new video dataset with attribute-based multi-label called Multi-Kinetics for evaluation, which contains various attribute labels (e.g. action, scene, object, etc.) related to human behavior. Extensive experiments demonstrate that our AAPM achieves the state-of-the-art performance in both attribute-based multi-label few-shot action recognition and single-label few-shot action recognition. The project and dataset are available at an anonymous account https://github.com/theAAPM/AAPM

Problem

• Enhances multi-label action recognition accuracy.
• Balances accuracy and robustness in recognition.
• Introduces new dataset for comprehensive evaluation.

Innovation

• Adaptive Attribute Prototype Model
• Text-Constrain Module integration
• Attribute Assignment Method