VL-MemKnG: Hybrid Memory with a Spatio-Temporal Knowledge Graph for Question Answering over Long Egocentric Navigation Trajectories

2026-06-15

AI Summary

This work addresses the challenge of question answering over long-horizon first-person navigation videos, which requires retrieving and integrating dispersed evidence across distant temporal points while preserving spatiotemporal and contextual consistency—a task where existing approaches suffer from limitations in either efficiency or coverage. To this end, we propose VL-MemKnG, a hybrid memory framework that uniquely integrates a structured spatiotemporal knowledge graph with persistent segment-level contextual memory, enabling efficient and accurate evidence aggregation and answer generation through a joint retrieval-reasoning module. We introduce WalkieKnowledgeT+, a new benchmark for this task, and demonstrate the superiority of our approach: it improves top-1 retrieval accuracy from 58% to 67% and Recall@1 from 34.50% to 40.55%, significantly outperforming state-of-the-art models such as Gemini 2.5 Pro and Qwen 3.5+.

Abstract

Answering navigation-relevant questions over long egocentric videos requires retrieving and organizing evidence distributed across distant temporal moments while maintaining spatial and contextual consistency. Although long-context vision--language models can achieve strong answer quality, they are computationally expensive for long trajectories and inefficient for repeated querying. Recent graph-based approaches such as VL-KnG address this challenge through persistent spatio-temporal knowledge graphs, but graph-centric retrieval alone may underrepresent broader temporal continuity and contextual cues. We present VL-MemKnG, a hybrid memory framework that extends VL-KnG by combining a spatio-temporal knowledge graph with persistent segment-level contextual memory. The knowledge graph captures structured relational information and long-range object associations, while segment-level memory preserves broader temporal context for long-horizon evidence retrieval. A hybrid retrieval-and-reasoning module jointly operates over both memory representations to produce evidence-grounded answers and temporally organized supporting evidence. We also introduce WalkieKnowledgeT+, an extension of WalkieKnowledge for long-horizon navigation-oriented video question answering. The benchmark includes temporally distributed reasoning tasks requiring evidence aggregation across multiple non-cooccurring moments. On WalkieKnowledgeT+, VL-MemKnG improves Top-1 retrieval accuracy from 58% to 67% and Recall@1 from 34.50% to 40.55%, outperforming all compared methods, including Gemini 2.5 Pro and Qwen 3.5+. The gains are particularly pronounced on temporal-global and temporally scattered aggregation questions, demonstrating the benefits of combining structured relational memory with segment-level contextual memory while maintaining efficient query-time inference.

Problem

• egocentric navigation
• long-horizon video question answering
• spatio-temporal reasoning
• evidence retrieval
• temporal context

Innovation

• hybrid memory
• spatio-temporal knowledge graph
• long-horizon video QA
• segment-level contextual memory
• egocentric navigation