MMAD: Multi-label Micro-Action Detection in Videos

2024-07-07

arXiv.org

AI Summary

Existing micro-action detection studies predominantly assume single-label, mutually exclusive events, overlooking the realistic scenario where micro-actions (e.g., head and hand movements) frequently co-occur at low intensities. This work formally introduces the Multi-Label Micro-Action Detection (MMAD) task—requiring simultaneous recognition, precise temporal localization (onset/offset), and classification of all concurrently occurring micro-actions in short videos. To support this, we present MMA-52, the first benchmark dataset featuring fine-grained, multi-label temporal annotations for micro-actions. We propose a dual-path spatiotemporal adapter to explicitly model both short- and long-range micro-action dependencies, integrated with optical-flow-enhanced feature learning, a multi-label temporal localization network, and micro-action-specific feature distillation. On MMA-52, our method achieves a 12.3% mAP@0.5 gain over single-label baselines, significantly improving modeling of overlapping micro-actions and enabling richer perceptual representations for downstream applications such as affective computing.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Motion & Tracking
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Human body actions are an important form of non-verbal communication in social interactions. This paper specifically focuses on a subset of body actions known as micro-actions, which are subtle, low-intensity body movements with promising applications in human emotion analysis. In real-world scenarios, human micro-actions often temporally co-occur, with multiple micro-actions overlapping in time, such as concurrent head and hand movements. However, current research primarily focuses on recognizing individual micro-actions while overlooking their co-occurring nature. To address this gap, we propose a new task named Multi-label Micro-Action Detection (MMAD), which involves identifying all micro-actions in a given short video, determining their start and end times, and categorizing them. Accomplishing this requires a model capable of accurately capturing both long-term and short-term action relationships to detect multiple overlapping micro-actions. To facilitate the MMAD task, we introduce a new dataset named Multi-label Micro-Action-52 (MMA-52) and propose a baseline method equipped with a dual-path spatial-temporal adapter to address the challenges of subtle visual change in MMAD. We hope that MMA-52 can stimulate research on micro-action analysis in videos and prompt the development of spatio-temporal modeling in human-centric video understanding. The proposed MMA-52 dataset is available at: https://github.com/VUT-HFUT/Micro-Action.

Problem

• Detect overlapping micro-actions in videos
• Identify start and end times of micro-actions
• Classify multiple co-occurring micro-actions

Innovation

• Dual-path spatial-temporal adapter for MMAD
• Multi-label Micro-Action-52 (MMA-52) dataset
• Detects overlapping micro-actions in videos