GLOSS: Group of LLMs for Open-Ended Sensemaking of Passive Sensing Data for Health and Wellbeing

2025-07-07

AI Summary

This study addresses the limitations of passive smartphone and wearable sensor data in high-level behavioral and contextual understanding—namely, insufficient openness for interpretive sensemaking and the difficulty of triangulating heterogeneous, multi-source temporal data. To this end, we propose the first LLM-augmented system supporting open-ended meaning construction and complex multimodal triangulation. Our approach integrates retrieval-augmented generation (RAG) with a multi-perspective temporal reasoning mechanism, enabled by a collaborative multi-LLM architecture that jointly performs semantic interpretation and contextual fusion. Evaluated across four real-world health and well-being use cases, the system achieves 87.93% accuracy and 66.19% consistency—substantially outperforming conventional RAG (29.31% accuracy, 52.85% consistency). Its core contribution lies in transcending the closed-loop RAG paradigm, establishing the first open, verifiable, and multi-model collaborative framework for temporal-sensor-data understanding.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The ubiquitous presence of smartphones and wearables has enabled researchers to build prediction and detection models for various health and behavior outcomes using passive sensing data from these devices. Achieving a high-level, holistic understanding of an individual's behavior and context, however, remains a significant challenge. Due to the nature of passive sensing data, sensemaking -- the process of interpreting and extracting insights -- requires both domain knowledge and technical expertise, creating barriers for different stakeholders. Existing systems designed to support sensemaking are either not open-ended or cannot perform complex data triangulation. In this paper, we present a novel sensemaking system, Group of LLMs for Open-ended Sensemaking (GLOSS), capable of open-ended sensemaking and performing complex multimodal triangulation to derive insights. We demonstrate that GLOSS significantly outperforms the commonly used Retrieval-Augmented Generation (RAG) technique, achieving 87.93% accuracy and 66.19% consistency, compared to RAG's 29.31% accuracy and 52.85% consistency. Furthermore, we showcase the promise of GLOSS through four use cases inspired by prior and ongoing work in the UbiComp and HCI communities. Finally, we discuss the potential of GLOSS, its broader implications, and the limitations of our work.

Problem

• Interpreting passive sensing data for health insights
• Overcoming barriers in sensemaking with domain expertise
• Enhancing open-ended analysis and multimodal data triangulation

Innovation

• Uses group of LLMs for open-ended sensemaking
• Performs complex multimodal data triangulation
• Outperforms Retrieval-Augmented Generation technique