Event-Aware Instructed Assistant for Referring Video Segmentation

2026-06-25

AI Summary

This work addresses a critical limitation in current video understanding approaches, which often treat videos as monolithic events and thereby overlook their inherent multi-event structure, leading to semantic ambiguity and hallucination. To overcome this, the authors propose an event-aware hierarchical understanding framework that decomposes a video into multiple atomic events via learnable event queries and aligns each event’s semantics through text guidance. Additionally, they introduce a hybrid object-pixel feature learning mechanism to enhance fine-grained object tracking. Built upon a multimodal large language model, the proposed method achieves substantial performance gains over existing techniques across five public benchmarks, with particularly notable improvements on referring expression video segmentation tasks.

Abstract

Existing referring video segmentation methods often treat a video as a single event consisting of multiple images, overlooking the fact that a video typically contains multiple distinct events. Under such a mechanism, the model needs to directly understand all the complex content in the video and text, which can easily lead to confusion and hallucinations. To address this issue, we propose to decompose a video to a set of simple events by learnable Event Query, and understand complex video content in an event-by-event, easy-to-understand manner. This is based on the observation that natural language expressions often divide a video into distinct, text-related segments, each representing a separate event within a compound event. We introduce EVIS, an Event-Aware Video Instructed Segmentation Assistant, which utilizes text-guided Event Queries to partition a video into simple events, extracting event-aware visual-text features to achieve a hierarchical understanding of the video. Additionally, we propose Object-Pixel-Hybrid Learning, which enables the MLLMs to track targets in long-term videos by integrating fine-grained pixel features with prior object queries. Extensive experimental results on 5 public benchmarks demonstrate EVIS's strong performance in addressing the referring video segmentation task.

Problem

• referring video segmentation
• video events
• event decomposition
• visual-text understanding
• temporal segmentation

Innovation

• Event-Aware Segmentation
• Referring Video Segmentation
• Event Query
• Object-Pixel-Hybrid Learning
• Hierarchical Video Understanding