ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions

2025-05-20

AI Summary

Current LLM agents exhibit limited proactive service capability in open-world environments, constrained by closed-loop perception and rule-based reasoning, leading to poor user intent understanding. This paper introduces the first context-aware, proactive LLM agent endowed with open-world sensory perception: it continuously acquires multimodal sensory data (e.g., video, audio) via wearable devices and jointly models cross-modal user intent with historical behavioral profiles to enable demand prediction and seamless tool invocation. We propose ContextAgentBench—the first benchmark tailored for proactive LLM agents—and a novel dual-path context-driven decision framework integrating sensory input and historical context. Experiments demonstrate that our approach achieves an 8.5% improvement in proactive prediction accuracy and a 6.0% gain in tool invocation accuracy on ContextAgentBench, covering nine daily-life scenarios and twenty distinct tools.

Technology Category

• Multiagent Systems: Modeling other Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts to enhance the proactive capabilities of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and the persona contexts from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants.

Problem

• Enhancing proactive LLM agents with sensory context awareness
• Improving user intent understanding via multi-dimensional sensory data
• Automating tool calls for unobtrusive proactive assistance

Innovation

• Uses multi-dimensional sensory data for intent understanding
• Combines sensory and historical data for proactive predictions
• Automatically calls tools for unobtrusive user assistance