Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents

2025-10-22

AI Summary

Existing evaluations of proactivity in LLM-based agents lack systematic, cross-source, long-horizon benchmarks. Method: We propose PROBE, the first formal framework that conceptualizes proactivity as a three-stage capability—problem discovery, bottleneck identification, and solution execution—and introduces a multi-stage evaluation pipeline supporting cross-context reasoning, long-term memory tracking, and action verification. Contribution/Results: PROBE is the first quantifiable, end-to-end benchmark for measuring agent proactivity. Empirical evaluation reveals that state-of-the-art models (e.g., GPT-5, Claude Opus-4.1) achieve only ~40% end-to-end success on realistic proactivity tasks, exposing fundamental deficiencies—including goal drift, context forgetting, and execution fragmentation. PROBE establishes a reproducible, scalable evaluation paradigm and identifies concrete avenues for improvement, thereby advancing rigorous, principled research on proactive LLM agents.

Technology Category

• Multiagent Systems: Distributed Problem Solving
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously. However, evaluating proactivity is challenging; current benchmarks are constrained to localized context, limiting their ability to test reasoning across sources and longer time horizons. To address this gap, we present PROBE (Proactive Resolution Of BottlEnecks). PROBE decomposes proactivity as a pipeline of three core capabilities: (1) searching for unspecified issues, (2) identifying specific bottlenecks, and (3) executing appropriate resolutions. We apply PROBE to evaluate leading LLMs and popular agentic frameworks, showing that even state-of-the-art models struggle to solve this benchmark. Computing our consistent measurements across frontier LLMs and agents, we find that the best end-to-end performance of 40% is achieved by both GPT-5 and Claude Opus-4.1. Additionally, we demonstrate the relative capabilities of each model and analyze mutual failure modes. Our results highlight the current limitations of autonomous action in agentic systems, and expose promising future research directions.

Problem

• Measuring proactive problem-solving capabilities in LLM agents
• Evaluating reasoning across multiple sources and time horizons
• Assessing autonomous issue identification and resolution in AI systems

Innovation

• PROBE framework measures proactive problem solving
• Decomposes proactivity into three core capabilities
• Evaluates LLMs through bottleneck identification pipeline