Distinguishing Visually Similar Actions: Prompt-Guided Semantic Prototype Modulation for Few-Shot Action Recognition

2025-12-22

AI Summary

Few-shot action recognition faces three key challenges: (1) dynamic motion features being corrupted by static background clutter, (2) difficulty distinguishing visually similar actions, and (3) a semantic alignment gap between multimodal (text-image) prototypes and visual query embeddings. To address these, we propose a Hierarchical Collaborative Motion Refinement module to suppress background noise and enhance discriminative motion representations. We further introduce a Semantic Prototype Modulation strategy coupled with a Prototype–Anchor Dual Modulation mechanism, enabling prompt-guided cross-modal alignment and support-query consistency optimization. Built upon the CLIP architecture, our framework integrates hierarchical spatiotemporal alignment, learnable textual prompt generation, and global semantic anchor guidance. Our method achieves state-of-the-art performance under 1-, 3-, and 5-shot settings on Kinetics, Something-Something v2, UCF101, and HMDB51. Ablation studies and visualization analyses comprehensively validate the effectiveness of each component.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Few-shot action recognition aims to enable models to quickly learn new action categories from limited labeled samples, addressing the challenge of data scarcity in real-world applications. Current research primarily addresses three core challenges: (1) temporal modeling, where models are prone to interference from irrelevant static background information and struggle to capture the essence of dynamic action features; (2) visual similarity, where categories with subtle visual differences are difficult to distinguish; and (3) the modality gap between visual-textual support prototypes and visual-only queries, which complicates alignment within a shared embedding space. To address these challenges, this paper proposes a CLIP-SPM framework, which includes three components: (1) the Hierarchical Synergistic Motion Refinement (HSMR) module, which aligns deep and shallow motion features to improve temporal modeling by reducing static background interference; (2) the Semantic Prototype Modulation (SPM) strategy, which generates query-relevant text prompts to bridge the modality gap and integrates them with visual features, enhancing the discriminability between similar actions; and (3) the Prototype-Anchor Dual Modulation (PADM) method, which refines support prototypes and aligns query features with a global semantic anchor, improving consistency across support and query samples. Comprehensive experiments across standard benchmarks, including Kinetics, SSv2-Full, SSv2-Small, UCF101, and HMDB51, demonstrate that our CLIP-SPM achieves competitive performance under 1-shot, 3-shot, and 5-shot settings. Extensive ablation studies and visual analyses further validate the effectiveness of each component and its contributions to addressing the core challenges. The source code and models are publicly available at GitHub.

Problem

• Improves temporal modeling by reducing static background interference in actions
• Enhances discrimination between visually similar action categories
• Bridges the modality gap between visual and textual prototypes for alignment

Innovation

• Aligns deep and shallow motion features to reduce static background interference
• Generates query-relevant text prompts to bridge visual-textual modality gaps
• Refines support prototypes and aligns queries with a global semantic anchor