PM-Bench: Evaluating Prospective Memory in LLM Agents

2026-07-14

AI Summary

This work addresses the challenge of prospective memory in large language model (LLM) agents—specifically, their ability to retain and execute delayed intentions in specified future contexts during ongoing tasks. To this end, it introduces PM-Bench, the first benchmark dedicated to evaluating this capability. Adapting the Virtual Week paradigm from cognitive science, PM-Bench constructs a controlled textual environment simulating a seven-day virtual week, enabling systematic assessment of agents’ capacity to maintain, trigger, and monitor delayed intentions amid routine activities. Experiments across eight state-of-the-art LLMs and diverse agent configurations demonstrate that PM-Bench poses a significant challenge: even the best-performing model (GPT-5.4) achieves only a 65.1% F1 score, and no single strategy proves universally effective across all models.

Abstract

A significant challenge in agentic AI is prospective memory: the ability to execute an intention at a specific future cue or state while other activities are ongoing. We introduce PM-Bench, a text-based benchmark for measuring prospective memory capabilities in modern LLM agents. Inspired by the Virtual Week paradigm from cognitive science, PM-Bench evaluates how well LLM agents maintain user intentions, execute delayed intentions, and monitor latent environment changes. Over the course of a simulated seven-day week, agents must continue an ongoing activity while deciding whether any deferred task is due. We compare eight state-of-the-art LLMs on PM-Bench under eight different agent configurations. PM-Bench proves challenging across all settings: the best method, a GPT-5.4 agent, reaches only 65.1\% F1 score under our evaluation. Furthermore, no single strategy for improving prospective memory dominates across models. We release PM-Bench as a controlled testbed for diagnosing these failures and developing training or inference-time interventions that support reliable prospective behavior.

Problem

• prospective memory
• LLM agents
• intention execution
• delayed tasks
• environment monitoring

Innovation

• prospective memory
• LLM agents
• PM-Bench
• Virtual Week
• delayed intention execution