ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems

2025-12-07

AI Summary

Existing LLM-based agents predominantly rely on explicit user instructions, imposing high operational and cognitive burdens. This work introduces the first end-to-end proactive LLM agent system that enables continuous contextual understanding and autonomous service initiation by jointly modeling multi-level environmental signals—visual, auditory, and behavioral—and user-specific cues. Methodologically, it proposes: (1) an on-demand hierarchical perception mechanism supporting a lightweight edge-cloud collaborative architecture; and (2) a context-aware proactive reasoning framework for real-time intent prediction and tool invocation from heterogeneous sensory inputs. Evaluated in real-world scenarios, the system achieves a 33.4% improvement in proactive intent prediction accuracy and a 16.8% gain in tool invocation F1-score, while significantly enhancing user satisfaction. This marks the first paradigm shift of LLM agents from reactive response to autonomous, proactive service delivery.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Model (LLM) agents are emerging to transform daily life. However, existing LLM agents primarily follow a reactive paradigm, relying on explicit user instructions to initiate services, which increases both physical and cognitive workload. In this paper, we propose ProAgent, the first end-to-end proactive agent system that harnesses massive sensory contexts and LLM reasoning to deliver proactive assistance. ProAgent first employs a proactive-oriented context extraction approach with on-demand tiered perception to continuously sense the environment and derive hierarchical contexts that incorporate both sensory and persona cues. ProAgent then adopts a context-aware proactive reasoner to map these contexts to user needs and tool calls, providing proactive assistance. We implement ProAgent on Augmented Reality (AR) glasses with an edge server and extensively evaluate it on a real-world testbed, a public dataset, and through a user study. Results show that ProAgent achieves up to 33.4% higher proactive prediction accuracy, 16.8% higher tool-calling F1 score, and notable improvements in user satisfaction over state-of-the-art baselines, marking a significant step toward proactive assistants. A video demonstration of ProAgent is available at https://youtu.be/pRXZuzvrcVs.

Problem

• Develops a proactive LLM agent system using sensory contexts
• Shifts from reactive to proactive assistance via on-demand perception
• Enhances prediction accuracy and user satisfaction in real-world applications

Innovation

• Proactive agent system with on-demand sensory context extraction
• Context-aware reasoner mapping sensory cues to user needs
• Implementation on AR glasses with edge server for real-world evaluation