ENTER: Event Based Interpretable Reasoning for VideoQA

2025-01-24

AI Summary

Existing video question answering (VQA) systems lack fine-grained visual understanding and traceable, interpretable reasoning paths. Method: We propose an event-graph-driven explainable VQA framework. First, videos are parsed into structured event graphs encoding temporal, causal, and hierarchical relationships. Second, a hierarchical iterative updating mechanism is designed within a graph neural network to fuse multi-granularity vision–language features; symbolic graph reasoning is then performed via executable code generation. Contribution/Results: This work introduces the first framework that jointly enables event-graph-guided hierarchical reasoning and low-level visual perception, achieving both strong interpretability and competitive performance. Experiments demonstrate state-of-the-art results among explainable models on NExT-QA, IntentQA, and EgoSchema, matching top end-to-end methods while providing complete, verifiable symbolic reasoning traces.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Question Answering

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this paper, we present ENTER, an interpretable Video Question Answering (VideoQA) system based on event graphs. Event graphs convert videos into graphical representations, where video events form the nodes and event-event relationships (temporal/causal/hierarchical) form the edges. This structured representation offers many benefits: 1) Interpretable VideoQA via generated code that parses event-graph; 2) Incorporation of contextual visual information in the reasoning process (code generation) via event graphs; 3) Robust VideoQA via Hierarchical Iterative Update of the event graphs. Existing interpretable VideoQA systems are often top-down, disregarding low-level visual information in the reasoning plan generation, and are brittle. While bottom-up approaches produce responses from visual data, they lack interpretability. Experimental results on NExT-QA, IntentQA, and EgoSchema demonstrate that not only does our method outperform existing top-down approaches while obtaining competitive performance against bottom-up approaches, but more importantly, offers superior interpretability and explainability in the reasoning process.

Problem

• Video Question Answering
• Reasoning Process
• Explainability

Innovation

• ENTER System
• Event Graph
• Dynamic Update