$π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

2026-05-14

AI Summary

Existing benchmarks struggle to evaluate intelligent assistants’ ability to proactively identify and respond to users’ implicit intentions in long-term, multi-turn interactions. To address this gap, this work introduces a novel benchmark comprising 100 multi-turn tasks spanning five user roles, enabling the first systematic joint assessment of proactivity and task completion. The benchmark more realistically simulates real-world scenarios where user needs gradually emerge by incorporating hidden intentions, inter-task dependencies, and cross-session continuity. It is accompanied by a comprehensive framework encompassing multi-turn task structures, role modeling, implicit intention injection, cross-session state tracking, and integrated evaluation metrics. Experimental results reveal that proactivity remains a significant challenge for current systems, with notable discrepancies between task completion rates and proactive performance, and further demonstrate that historical interactions critically influence subsequent recognition of proactive intent.

Abstract

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $π$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $π$-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.

Problem

• proactive assistance
• personal assistant agents
• hidden intents
• long-horizon workflows
• multi-turn interactions

Innovation

• proactive assistance
• long-horizon workflows
• hidden user intents
• multi-turn tasks
• agent benchmarking