OpenLifelogQA: An Open-Ended Multi-Modal Lifelog Question-Answering Dataset

2025-08-05

AI Summary

Existing lifelog question answering (QA) research is constrained by small-scale, synthetic, or closed-domain datasets, lacking large-scale, open-ended, multimodal QA resources grounded in real-world scenarios. To address this gap, we introduce LifelogQA—the first large-scale, open-domain lifelog QA dataset derived from 18 months of authentic wearable-device data—comprising 14,187 high-quality, human-annotated QA pairs spanning diverse question types and difficulty levels. LifelogQA enables interactive exploration of personal daily memories and bridges the critical shortage of real-world data for memory-augmented applications. Benchmarking on LLaVA-NeXT-Interleave 7B demonstrates both practical utility and substantial challenge: BERTScore reaches 89.7%, ROUGE-L attains 25.87%, and LLM Score achieves 3.9665, confirming its viability for advancing multimodal lifelog understanding and retrieval.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Lifelogging refers to the process of passively collecting, storing, and analysing personal daily life data using wearable devices. This data can support applications in memory preservation and enhancement. For example, using an ask-and-answer strategy, question-answering (QA) on lifelog data opens an interactive and interesting way to explore memorable events and insights into daily life. However, research resources for QA on lifelog data are limited to small-sized or synthetic QA datasets. In this paper, we present a novel lifelog QA dataset called OpenLifelogQA, building upon an 18-month lifelog dataset. Our dataset focuses on an open-ended and practical QA with real-world application in daily lifelog usage. We construct 14,187 pairs of Q&A with diverse types and difficulty levels. A baseline experiment is reported for this dataset with competitive average performance of 89.7% BERT Score, 25.87% ROUGE-L and 3.9665 LLM Score from LLaVA-NeXT-Interleave 7B model. We release this Q&A dataset to the research community to support new research into lifelog technologies, such as enabling personal chat-based assistants for lifelog data to become a reality.

Problem

• Lack of large-scale real-world lifelog QA datasets
• Need for open-ended QA in daily lifelog applications
• Limited research resources for lifelog memory enhancement

Innovation

• Multi-modal lifelog QA dataset creation
• Diverse open-ended Q&A pairs construction
• Baseline performance with advanced LLM