The Demon is in Ambiguity: Revisiting Situation Recognition with Single Positive Multi-Label Learning

2025-08-29

AI Summary

Existing verb classification methods formulate visual event recognition as a single-label task, failing to address the inherent semantic ambiguity in images—where a single image may reasonably correspond to multiple verbs. This work is the first to characterize verb classification in situational recognition as a single-positive multi-label problem and proposes the first systematic solution. Our approach comprises three key contributions: (1) constructing the first comprehensive benchmark supporting multi-label evaluation; (2) designing Graph-Enhanced Verb MLP (GE-VerbMLP), which leverages graph neural networks to model semantic relationships among verbs; and (3) incorporating adversarial training to refine decision boundaries. Evaluated on real-world datasets, our method achieves over a 3% improvement in mean Average Precision (mAP) while maintaining competitive Top-1 and Top-5 accuracy. This work establishes a new paradigm for visual event understanding by explicitly addressing multi-verb semantics in contextual scenes.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Context recognition (SR) is a fundamental task in computer vision that aims to extract structured semantic summaries from images by identifying key events and their associated entities. Specifically, given an input image, the model must first classify the main visual events (verb classification), then identify the participating entities and their semantic roles (semantic role labeling), and finally localize these entities in the image (semantic role localization). Existing methods treat verb classification as a single-label problem, but we show through a comprehensive analysis that this formulation fails to address the inherent ambiguity in visual event recognition, as multiple verb categories may reasonably describe the same image. This paper makes three key contributions: First, we reveal through empirical analysis that verb classification is inherently a multi-label problem due to the ubiquitous semantic overlap between verb categories. Second, given the impracticality of fully annotating large-scale datasets with multiple labels, we propose to reformulate verb classification as a single positive multi-label learning (SPMLL) problem - a novel perspective in SR research. Third, we design a comprehensive multi-label evaluation benchmark for SR that is carefully designed to fairly evaluate model performance in a multi-label setting. To address the challenges of SPMLL, we futher develop the Graph Enhanced Verb Multilayer Perceptron (GE-VerbMLP), which combines graph neural networks to capture label correlations and adversarial training to optimize decision boundaries. Extensive experiments on real-world datasets show that our approach achieves more than 3% MAP improvement while remaining competitive on traditional top-1 and top-5 accuracy metrics.

Problem

• Addressing verb ambiguity in situation recognition as multi-label problem
• Proposing single positive multi-label learning for verb classification
• Developing graph-based model to capture label correlations in events

Innovation

• Single Positive Multi-Label Learning for verb classification
• Graph Enhanced Verb MLP with neural networks
• Adversarial training optimizes decision boundaries