Detection-Fusion for Knowledge Graph Extraction from Videos

2024-12-30

AI Summary

To address the limitations of natural language (NL) descriptions in video understanding—including semantic abstraction, evaluation difficulty, and weak cross-lingual alignment—this paper proposes the first end-to-end Video-to-Knowledge Graph (Video-to-KG) generation framework. Methodologically, it adopts a two-stage detect-then-fuse paradigm: first localizing entity pairs in videos, then jointly modeling their semantic relations. Crucially, it introduces an external knowledge graph injection mechanism to enhance logical consistency and commonsense plausibility of the generated KG. Technically, the framework integrates multi-stage deep detection, entity-pair identification, relation classification, and knowledge-augmented encoding. Evaluated on a standard video KG benchmark, it significantly outperforms NL-based baselines, achieving an 18.7% F1-score improvement. The generated KGs support structured querying, cross-lingual mapping, and automated evaluation—establishing a novel, interpretable, and computationally tractable paradigm for video semantic understanding.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Description Logics
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

One of the challenging tasks in the field of video understanding is extracting semantic content from video inputs. Most existing systems use language models to describe videos in natural language sentences, but this has several major shortcomings. Such systems can rely too heavily on the language model component and base their output on statistical regularities in natural language text rather than on the visual contents of the video. Additionally, natural language annotations cannot be readily processed by a computer, are difficult to evaluate with performance metrics and cannot be easily translated into a different natural language. In this paper, we propose a method to annotate videos with knowledge graphs, and so avoid these problems. Specifically, we propose a deep-learning-based model for this task that first predicts pairs of individuals and then the relations between them. Additionally, we propose an extension of our model for the inclusion of background knowledge in the construction of knowledge graphs.

Problem

• Video Understanding
• Natural Language Processing
• Description Translation

Innovation

• Knowledge Graph
• Video Information Fusion
• Multimodal Learning