Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation

2026-06-19

AI Summary

This work addresses the challenge of policy failure in long-horizon, contact-rich robotic manipulation tasks caused by partial observability—where a single visual observation often lacks sufficient contextual information about ongoing actions. To overcome this limitation, the authors propose the Compressed Action Memory Policy (CAMP), which leverages self-supervised learning to compress historical action sequences into an implicit behavioral memory. This memory is seamlessly integrated into an end-to-end policy, enabling effective utilization of past experiences—including failures—without requiring additional supervision or external intervention. Extensive experiments across four real-world robotic platforms and two novel simulation benchmarks demonstrate that CAMP significantly outperforms current state-of-the-art methods, highlighting its effectiveness and generalization capability in complex manipulation tasks.

Abstract

Long horizon, contact-rich manipulation is inherently partially observable. This is as a single visual observation rarely captures a robot's full action context, including prior attempts, interactions, or progress. Consequently, standard visuomotor policies or vision-language-action models are prone to struggle in such tasks due to a lack of memory. To address this, we introduce Compressed Action Memory Policy (CAMP) based on the insight that a robot's own action history serves as a highly informative, self-supervised signal, enabling the policy to learn a robust, compact history representation. In our approach, we train a memory module to maintain a compressed representation of past actions, forcing it to encode a latent behavioral memory of all the robot's past interactions that can then be used to better contextualize future actions. This allows our approach to implicitly track generalized task progress and learn from failed attempts without any additional supervision, or external oversight. We evaluate CAMP across four real-robot setups and two novel simulation benchmarks: Memory-T-Bench and Memory-Manip-Bench. By demonstrating substantial gains over state-of-the-art baselines, CAMP is, to our knowledge, the first policy to demonstrate substantial success on contact-rich partially observable manipulation tasks purely through learned memory.

Problem

• partial observability
• behavioral memory
• object manipulation
• action history
• visuomotor policy

Innovation

• Compressed Action Memory
• Partially Observable Manipulation
• Behavioral Memory
• Self-supervised Learning
• Contact-rich Tasks