EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory

2026-08-12

AI Summary

Current first-person long-term memory question-answering systems suffer from limited performance due to context-poor indexing and retrieval mechanisms that overlook temporal intent. To address this, this work proposes the EgoCITE framework, which constructs atomic memory indices enriched with local multimodal context and organizes them into a multi-granularity, multi-view memory structure. It further introduces a joint semantic- and temporal-aware evidence retrieval mechanism. EgoCITE is the first approach to unify context-enhanced indexing with temporal-aware retrieval within a single modeling framework, significantly improving both accuracy and efficiency in egocentric long-term memory QA. Evaluated on EgoLifeQA, EgoMem, and EgoR1-Bench, the method outperforms existing baselines by 4.4–14.2% in accuracy while reducing inference cost by up to 36×.

Abstract

Long-horizon egocentric memory transforms continuous first-person video and audio into a searchable record of past experiences. We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent. To address both bottlenecks, we introduce EgoCITE (Egocentric Context-augmented Indexing and Time-aware Evidence retrieval), a long-horizon agentic memory framework for egocentric QA. EgoCITE comprises three components. EgoScheme uses local multimodal context to turn fragmentary video captions and speech transcripts into self-contained atomic memory indices. EgoIndex organizes complementary action, activity, utterance, and conversation representations into searchable multi-view memory indices at multiple granularities. EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved evidence. We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench in terms of answer accuracy and target-event retrieval alignment. EgoCITE improves accuracy over agentic memory baselines by at least 4.4--14.2\% while achieving 36$\times$ lower cost than long-context LLM agents.

Problem

• egocentric memory
• context-poor indexing
• temporal intent
• long-horizon retrieval
• agentic search

Innovation

• context-augmented indexing
• time-aware retrieval
• egocentric memory
• multi-view representation
• agentic QA