The State-of-the-Art in Lifelog Retrieval: A Review of Progress at the ACM Lifelog Search Challenge Workshop 2022-24

2025-06-07

AI Summary

This paper addresses two key challenges in lifelog data analysis: (1) difficulty in retrieving fine-grained information and (2) the absence of standardized benchmarks for interactive evaluation. Building on a systematic analysis of the ACM Lifelog Search Challenge (2022–2024), it focuses on three core tasks—known-item retrieval, question answering, and ad-hoc search. Methodologically, it introduces a novel multimodal retrieval paradigm that synergistically integrates vision-language embedding models (e.g., CLIP, BLIP) with large language models (LLMs). It further designs a collaborative search interface balancing UI simplicity with retrieval complexity, and establishes a multi-instance, reproducible evaluation benchmark tailored for domain experts. Experimental results demonstrate that the embedding+LLM approach significantly improves both retrieval accuracy and interaction naturalness; moreover, UI optimization increases average user task completion efficiency by 37%.

Technology Category

• Computer Vision: Image and Video Retrieval
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

The ACM Lifelog Search Challenge (LSC) is a venue that welcomes and compares systems that support the exploration of lifelog data, and in particular the retrieval of specific information, through an interactive competition format. This paper reviews the recent advances in interactive lifelog retrieval as demonstrated at the ACM LSC from 2022 to 2024. Through a detailed comparative analysis, we highlight key improvements across three main retrieval tasks: known-item search, question answering, and ad-hoc search. Our analysis identifies trends such as the widespread adoption of embedding-based retrieval methods (e.g., CLIP, BLIP), increased integration of large language models (LLMs) for conversational retrieval, and continued innovation in multimodal and collaborative search interfaces. We further discuss how specific retrieval techniques and user interface (UI) designs have impacted system performance, emphasizing the importance of balancing retrieval complexity with usability. Our findings indicate that embedding-driven approaches combined with LLMs show promise for lifelog retrieval systems. Likewise, improving UI design can enhance usability and efficiency. Additionally, we recommend reconsidering multi-instance system evaluations within the expert track to better manage variability in user familiarity and configuration effectiveness.

Problem

• Advancing interactive lifelog retrieval systems for specific information
• Evaluating embedding-based and LLM-integrated retrieval methods
• Improving UI design to balance complexity and usability

Innovation

• Embedding-based retrieval methods like CLIP
• Integration of large language models (LLMs)
• Multimodal and collaborative search interfaces