AwareLLM: A Proactive Multimodal Ecosystem for Personalized Human-AI Collaboration to Enhance Productivity

2026-05-10

AI Summary

This work addresses the limitation of current AI assistants, which rely solely on predefined preferences and chat history while lacking awareness of users’ psychological and physiological states, thereby hindering truly personalized support. The paper proposes a proactive multimodal human-AI collaboration system that, for the first time, deeply integrates first-person vision, pupillometry, eye tracking, posture detection, and heart rate signals with a large language model to enable real-time inference of user state and timely interventions. This approach shifts the paradigm from passive responsiveness to active adaptation. User studies demonstrate that, compared to a standard LLM-based assistant, the proposed system significantly improves task performance, reduces cognitive load and mental fatigue, and delivers interventions perceived as timely and effective, thereby enhancing users’ confidence and engagement in their tasks.

Abstract

Information workers' productivity is significantly influenced by their cognitive states and physiological responses. AI assistants such as ChatGPT, Copilot, and others have become integral components of knowledge-intensive workplaces. These AI assistants utilize pre-defined user preferences and chat interaction histories, thus confining themselves to reactive exchanges, lacking sufficient adaptability. Consequently, they fail to cater to individual user preferences and are unable to adapt to their psychophysiological states, diminishing potential productivity gains. To bridge this gap, we introduce AwareLLM, a novel multimodal framework that integrates egocentric vision, pupillometry, eye-gaze tracking, posture detection, heart activity, and the inferencing capabilities of large language models (LLMs) to create a proactive and context-aware ecosystem. AwareLLM dynamically adapts to users' psychophysiological states while analyzing temporal patterns and behavioral tendencies to provide personalized and timely interventions. We evaluated AwareLLM through a user study with 20 participants, comparing it to a standard LLM assistant across multiple tasks. Our results show statistically significant improvements in task performance, along with reductions in cognitive fatigue and mental demand. Participants described AwareLLM's personalized interventions as timely and relevant, helping them boost their confidence and deepen engagement with their work. AwareLLM opens new avenues for Human-AI collaboration where technology adapts to our needs rather than us adhering to technological constraints.

Problem

• AI assistants
• psychophysiological states
• personalization
• proactive interaction
• productivity

Innovation

• multimodal sensing
• psychophysiological adaptation
• proactive AI
• context-aware LLM
• personalized human-AI collaboration