EgoLife: Towards Egocentric Life Assistant

2025-03-05

AI Summary

To address key bottlenecks in egocentric video understanding for life-assistant applications—namely, long-horizon event retrieval and personalized recommendation generation—this paper introduces EgoButler. We construct EgoLife, the first large-scale, multimodal egocentric dataset tailored to daily living, comprising 300 hours of first-person videos, complementary third-person footage, and dense spatiotemporal annotations. We further propose EgoLifeQA, a life-oriented, long-horizon question-answering benchmark. Methodologically, we develop EgoGPT, a multimodal pre-trained model integrating audio-visual synchronization, cross-view identity association, and self-supervised learning; and EgoRAG, a retrieval-augmented framework enabling precise QA over multi-hour videos. Experiments show state-of-the-art performance on egocentric understanding tasks, with EgoRAG significantly enhancing cross-temporal reasoning. We publicly release the EgoLife dataset, EgoLifeQA benchmark, and model weights to advance embodied AI for everyday life.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities - including discussions, shopping, cooking, socializing, and entertainment - using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations. To address the key technical challenges of (1) developing robust visual-audio models for egocentric data, (2) enabling identity recognition, and (3) facilitating long-context question answering over extensive temporal information, we introduce EgoButler, an integrated system comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.

Problem

• Develop AI-powered wearable glasses for personal efficiency enhancement.
• Create a comprehensive dataset for egocentric, interpersonal, and multimodal daily life.
• Address challenges in visual-audio modeling, identity recognition, and long-context question answering.

Innovation

• AI-powered wearable glasses for egocentric data capture
• EgoLife Dataset: 300-hour multimodal daily life dataset
• EgoButler: Integrated system with EgoGPT and EgoRAG