HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy

2025-10-01

AI Summary

Existing vision-language-action (VLA) models neglect historical context, limiting their performance on robot manipulation tasks with strong temporal dependencies. To address this, we propose HAMLET—a novel framework that introduces temporal contrastive learning for initializing temporal token representations and incorporates a lightweight memory module to efficiently model history-aware information and enable end-to-end action prediction. HAMLET is plug-and-play and highly extensible, requiring no modifications to the backbone architecture. Evaluated on GR00T N1.5, it achieves an average success rate of 76.4% (+47.2 percentage points) on history-dependent tasks. Moreover, it significantly outperforms baselines on the RoboCasa and LIBERO benchmarks. These results demonstrate that explicit history-aware modeling is critical for advancing VLA performance, establishing a new state-of-the-art in temporally grounded robotic policy learning.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Inherently, robotic manipulation tasks are history-dependent: leveraging past context could be beneficial. However, most existing Vision-Language-Action models (VLAs) have been designed without considering this aspect, i.e., they rely solely on the current observation, ignoring preceding context. In this paper, we propose HAMLET, a scalable framework to adapt VLAs to attend to the historical context during action prediction. Specifically, we introduce moment tokens that compactly encode perceptual information at each timestep. Their representations are initialized with time-contrastive learning, allowing them to better capture temporally distinctive aspects. Next, we employ a lightweight memory module that integrates the moment tokens across past timesteps into memory features, which are then leveraged for action prediction. Through empirical evaluation, we show that HAMLET successfully transforms a state-of-the-art VLA into a history-aware policy, especially demonstrating significant improvements on long-horizon tasks that require historical context. In particular, on top of GR00T N1.5, HAMLET achieves an average success rate of 76.4% on history-dependent real-world tasks, surpassing the baseline performance by 47.2%. Furthermore, HAMLET pushes prior art performance from 64.1% to 66.4% on RoboCasa Kitchen (100-demo setup) and from 95.6% to 97.7% on LIBERO, highlighting its effectiveness even under generic robot-manipulation benchmarks.

Problem

• Enabling Vision-Language-Action models to incorporate historical context for robotic manipulation
• Addressing limitations of current VLAs that ignore preceding temporal information
• Improving performance on long-horizon tasks requiring memory of past observations

Innovation

• Introduces moment tokens for compact perceptual encoding
• Uses time-contrastive learning to capture temporal distinctiveness
• Integrates lightweight memory module for historical context