HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents

2026-06-15

AI Summary

This work addresses the credit assignment bias in long-horizon agents, where causal entanglement during memory writing often leads to the erroneous deletion of useful information or retention of irrelevant content. To mitigate this issue, the authors propose HiMPO, a framework that evaluates the local utility of memory updates through post-hoc information analysis and employs a bounded retrospective filter to retroactively reweight memory credits based on downstream task outcomes. By decoupling memory-writing attribution from other behavioral decisions, HiMPO integrates memory-specific advantage estimation, reinforcement learning, and compressed memory mechanisms to significantly enhance memory credit accuracy. Experimental results demonstrate that HiMPO outperforms strong baselines in open-domain tasks and compressed-memory question answering, effectively alleviating blame leakage while maintaining high context compression efficiency.

Abstract

Long-horizon agents rely on memory mechanisms to compress interaction history, but optimizing memory writing faces a distinct credit assignment challenge: a memory update may be rewarded or penalized due to downstream tool failures, noisy observations, or reasoning errors rather than its own contribution. This causally entangled credit can lead agents to discard useful evidence or preserve irrelevant information. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework for assigning less-entangled credit to memory-writing actions in long-horizon agents. HiMPO first estimates the local utility of a memory update by comparing the task-relevant information recoverable from the previous and updated memories under the same pre-write state. It then uses hindsight relevance as a bounded retrospective filter that attenuates memory credit when local utility is not supported by the target outcome. The resulting memory-specific advantage is applied only to memory tokens, while trajectory-level rewards optimize the rest of the agent behavior. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves over strong memory-based and RL-based baselines while preserving compressed-context efficiency. Controlled interventions further show that HiMPO reduces blame leakage from tool-induced errors and improves attribution fidelity of memory updates.

Problem

• credit assignment
• memory policy
• long-horizon agents
• hindsight relevance
• memory optimization

Innovation

• credit assignment
• memory policy optimization
• hindsight relevance
• long-horizon agents
• less-entangled credit