Companion-style QA Assistance in Ego-Vision

2026-09-06

AI Summary

为了解决日常第一人称视角下连续视频问答中的指示代词和连贯问题，本文提出了MyBuddy系统，采用多模态思维链推理机制来提高回答准确性。

Abstract

AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video. BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, egocentric videos. It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks: ego-deictic expressions and interactively chained questions (e.g.,"Where is it?","How to get there?"). These require models to infer a user's in-situation intent by resolving visual pronouns in the context of egocentric visual and QA contents, with both grounded in a long-form streaming setting. To tackle the challenges, we propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reasoning mechanism to infer the final answer based on the historical QA and visual content. An additional question filter and multi-level memory are designed to facilitate efficient QA and visual information retrieval under streaming QA settings. Experiments show that MyBuddy significantly enhances the performance of foundation models on BuddyVQA. Moreover, these gains generalize to other streaming and common video QA benchmarks, demonstrating the applicability and effectiveness of our approach. Our code and dataset are available at https://github.com/QHUni/BuddyVQA

Problem

• Ego-Vision
• Question Answering
• Ego-deictic Expressions
• Interactively Chained Questions

Innovation

• Ego-Vision
• Companion-Style QA
• Multimodal Chain-of-Thought Reasoning
• Streaming Video