Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering

2026-08-18

AI Summary

为解决视觉语言模型在实体场景中资源受限的问题，提出了一种基于记忆树指导的关键帧选择方法，以提高3D问答效率。

Abstract

Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D

Problem

• visual search
• key frame retrieval
• embodied scenarios
• computational and memory resources
• Vision Language Model (VLM)

Innovation

• memory tree
• key frame selection
• 3D question answering
• MemTree3D