A Baseline Study and Benchmark for Few-Shot Open-Set Action Recognition with Feature Residual Discrimination

2026-03-04

AI Summary

This work addresses the challenge of few-shot action recognition in open-world scenarios, where the closed-set assumption fails and models struggle to reject unknown actions. To this end, we establish the first benchmark for few-shot open-set action recognition and propose a Feature Residual Discriminator (FR-Disc) architecture. FR-Disc extends open-set recognition from images to videos by modeling spatiotemporal feature residuals to effectively distinguish between known and unknown actions. The method maintains high closed-set recognition accuracy while significantly improving open-set generalization. Extensive experiments on five standard datasets demonstrate that FR-Disc substantially outperforms existing approaches, setting a new state-of-the-art performance for this task.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Algorithmic accountability and transparency on the web
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Few-Shot Action Recognition (FS-AR) has shown promising results but is often limited by a closed-set assumption that fails in real-world open-set scenarios. While Few-Shot Open-Set (FSOS) recognition is well-established for images, its extension to spatio-temporal video data remains underexplored. To address this, we propose an architectural extension based on a Feature-Residual Discriminator (FR-Disc), adapting previous work on skeletal data to the more complex video domain. Extensive experiments on five datasets demonstrate that while common open-set techniques provide only marginal gains, our FR-Disc significantly enhances unknown rejection capabilities without compromising closed-set accuracy, setting a new state-of-the-art for FSOS-AR. The project website, code, and benchmark are available at: https://hsp-iit.github.io/fsosar/.

Problem

• Few-Shot Action Recognition
• Open-Set Recognition
• Video Action Recognition
• Spatio-Temporal Data

Innovation

• Few-Shot Open-Set Action Recognition
• Feature-Residual Discriminator
• Unknown Rejection
• Video Action Recognition
• Open-Set Recognition