OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation

2026-08-14

AI Summary

This study addresses the closed-set limitations and high training costs of open-vocabulary approaches in dynamic scene graph generation by proposing the first end-to-end open-vocabulary framework. The method integrates a spatiotemporal backbone with a distillation-free vision-language alignment module, achieving efficient recognition through triplet feature extraction and adaptive decision boundary learning. Furthermore, a rigorous open-vocabulary evaluation benchmark is established. Experimental results demonstrate that the proposed model significantly outperforms existing baselines in zero-shot Recall@K by 10.0–20.4 percentage points while maintaining state-of-the-art closed-set performance. These findings indicate that the framework effectively overcomes bottlenecks in long-tail concept recognition, offering a robust solution for open-vocabulary dynamic scene understanding without compromising accuracy on head categories.

Abstract

Dynamic scene graphs (DSGs) capture spatio-temporal interactions across videos as $\langle$subject, predicate, object$\rangle$ triplets, and underpin downstream tasks such as video captioning, video question answering, and action analysis. However, end-to-end dynamic scene graph generation (DSGG) methods are closed-set: they recognize only objects and predicates from a fixed training vocabulary and struggle with the long-tailed distribution of rare concepts, severely limiting their real-world applicability. Existing open-vocabulary models typically inherit pretrained large language models, resulting in multi-stage training and inference with substantial cost. We introduce OvDSGG, the first end-to-end framework for open-vocabulary DSGG. OvDSGG builds on top of an open-vocabulary Spatial Backbone and a Temporal Backbone; we further propose a Triplet Feature Extraction Module that bridges them, and a Visual-Language Alignment Module that preserves open-vocabulary recognition by learning an adaptive decision boundary in the joint visual-language feature space, without expensive knowledge distillation in existing methods. We further introduce a rigorous open-vocabulary DSGG benchmark adapted from Action Genome, with disjoint Base/Novel splits for both objects and predicates. OvDSGG significantly outperforms open-vocabulary baselines across all metrics, with zero-shot Recall@$K$ scores 10.0--20.4 percentage point higher than the next-best baseline, while on closed-set DSGG remaining competitive with state-of-the-art models. Code and benchmark are publicly available at https://github.com/jhelsby/OvDSGG/.

Problem

• Dynamic Scene Graph Generation
• Open-Vocabulary
• End-to-End
• Long-tailed Distribution

Innovation

• End-to-End Open-Vocabulary DSGG
• Triplet Feature Extraction Module
• Visual-Language Alignment Module
• Adaptive Decision Boundary
• Open-Vocabulary Benchmark