One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition

2026-04-25

AI Summary

This work addresses the challenge of entity identity inconsistency in video situation recognition, where entities often assume varying roles across multiple events and shots, thereby undermining the coherence of action understanding and visual grounding. To resolve this, we propose CineMEC, the first multimodal entity coreference resolution framework tailored for video situation recognition. CineMEC jointly models event role mentions in text and visual entity clusters in video to enforce cross-modal entity consistency without requiring explicit grounding supervision. Our approach employs a multi-stage fusion architecture, event-role grouping alignment, and self-supervised coreference learning within an end-to-end trainable framework, simultaneously enhancing video captioning and visual grounding performance. Evaluated on an extended VidSitu dataset, CineMEC achieves a 2.5% improvement in CIDEr, a 7% gain in LEA, and a substantial 18% increase in HOTA for visual grounding.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Video Situation Recognition (VidSitu) addresses the challenging problem of "who did what to whom, with what, how, and where" in a video. It tests thorough video understanding by requiring identification of salient actions and associated short descriptions for event roles across multiple events. Grounding with VidSitu requires spatio-temporal localization of key entities across shots and varied appearances. We posit that coherent video understanding requires consistent identification of entities that play different roles. We propose Multimodal Entity Coreference (MEC) to unite entity descriptions in text with grounding across the video. Towards this, we introduce CineMEC, a multi-stage approach that unites event role mention groups with visual clusters of entities, without explicit grounding supervision during training. Our approach is designed to exploit the synergy between visual grounding and captioning, where improving one influences the other and vice versa. For evaluation, we extend the VidSitu dataset with grounding annotations. While previous work focuses primarily on descriptions, CineMEC improves consistency across both: captioning (+2.5% CIDEr, +7% LEA) and visual grounding (+18% HOTA).

Problem

• Video Situation Recognition
• Entity Coreference
• Multimodal Grounding
• Role Consistency
• Spatio-temporal Localization

Innovation

• Multimodal Entity Coreference
• Video Situation Recognition
• Visual Grounding
• CineMEC
• Entity Consistency