ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data

2026-02-04

AI Summary

This work addresses a critical limitation in existing proactive agent research, which predominantly relies on synthetic data and lacks contextual grounding in real-world user workflows, thereby hindering effective modeling of both intervention timing and content. To bridge this gap, the authors introduce ProAgentBench—the first benchmark for proactive agents based on authentic long-term user interactions. The framework decouples proactive assistance into two subtasks—timing prediction and assistive content generation—within a hierarchical task structure. A privacy-compliant dataset is constructed to preserve contextual integrity while capturing high-burst behavioral patterns (B=0.787). Experimental results demonstrate that models trained on real interaction data significantly outperform synthetic-data baselines, and that incorporating long-term context substantially improves intervention timing accuracy.

Technology Category

• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Proactive agents that anticipate user intentions without explicit prompts represent a significant evolution in human-AI interaction, promising to reduce cognitive load and streamline workflows. However, existing datasets suffer from two critical deficiencies: (1) reliance on LLM-synthesized data that fails to capture authentic human decision-making patterns, and (2) focus on isolated tasks rather than continuous workflows, missing the pre-assistance behavioral context essential for learning proactive intervention signals. To address these gaps, we introduce ProAgentBench, a rigorous benchmark for proactive agents in working scenarios. Our contributions include: (1) a hierarchical task framework that decomposes proactive assistance into timing prediction and assist content generation; (2) a privacy-compliant dataset with 28,000+ events from 500+ hours of real user sessions, preserving bursty interaction patterns (burstiness B=0.787) absent in synthetic data; and (3) extensive experiments that evaluates LLM- and VLM-based baselines. Numerically, we showed that long-term memory and historical context significantly enhance prediction accuracy, while real-world training data substantially outperforms synthetic alternatives. We release our dataset and code at https://anonymous.4open.science/r/ProAgentBench-6BC0.

Problem

• Proactive Agents
• Long-term User Context
• VLM Annotation
• Privacy Protection
• Human-Computer Interaction

Innovation

• Proactive Agents
• Long-term User Context
• VLM Annotation
• Privacy Protection
• Human-Computer Interaction