🤖 AI Summary
Existing visual relationship forecasting (VRF) methods suffer from label noise, weak semantic correlation among relationships, and insufficient discriminative capability for visually similar or temporally invariant relations. This paper introduces video-level VRF—a novel task that forecasts the interaction relationship between subject-object pairs over T future frames *without visual input*, conditioned solely on the preceding H frames. To support this task, we construct the first benchmark datasets—VRF-AG and VRF-VidOR—featuring spatiotemporally localized relationship annotations. We propose the Graph Convolutional Network–Transformer (GCT) hybrid architecture, which jointly models object dynamics via spatiotemporal graph convolutions and captures long-range dependencies via Transformers, augmented with relation-aware and multi-granularity (object-level and frame-level) dependency modeling. Experiments demonstrate that GCT significantly outperforms existing sequential modeling approaches on both datasets, validating its effectiveness and generalizability for long-horizon relational reasoning.
📝 Abstract
Real-world scenarios often require the anticipation of object interactions in unknown future, which would assist the decision-making process of both humans and agents. To meet this challenge, we present a new task named Visual Relationship Forecasting (VRF) in videos to explore the prediction of visual relationships in a reasoning manner. Specifically, given a subject-object pair with H existing frames, VRF aims to predict their future interactions for the next T frames without visual evidence. To evaluate the VRF task, we introduce two video datasets named VRF-AG and VRF-VidOR, with a series of spatio-temporally localized visual relation annotations in a video. These two datasets densely annotate 13 and 35 visual relationships in 1923 and 13447 video clips, respectively. In addition, we present a novel Graph Convolutional Transformer (GCT) framework, which captures both object-level and frame-level dependencies by spatio-temporal Graph Convolution Network and Transformer. Experimental results on both VRF-AG and VRF-VidOR datasets demonstrate that GCT outperforms the state-of-the-art sequence modelling methods on visual relationship forecasting.
