KEMO: Event-Driven Keyframe Memory for Long-Horizon Robot Manipulation with VLA Policies

2026-06-22

AI Summary

This work addresses the challenge of ambiguous action decisions in long-horizon robotic manipulation, where visually similar observations across distinct task phases can mislead policy inference. To mitigate this, the authors propose KEMO—a lightweight, plug-and-play memory framework that automatically selects event-driven keyframes by fusing robot kinematics with visual filtering, thereby constructing compact temporal memory tokens. KEMO innovatively integrates a gated residual fusion mechanism with a keyframe-aligned loss weighting strategy and employs cross-attention to effectively merge memory tokens with current visual features, enhancing vision-language-action (VLA) policies. Evaluated on real-world dual-arm tasks, KEMO improves task success rate by 23.6% and phase completion rate by 34.1% over memory-free baselines.

Abstract

Long-horizon robot manipulation remains challenging because similar observations may occur at different execution stages, while the appropriate action depends on previously completed operations. Memory can address this ambiguity by enabling policies to infer task progress from execution history. However, existing memory-augmented approaches often either retain dense histories that require compression or rely primarily on recent context that may discard earlier task-relevant events. In this work, we propose propose KEMO, a lightweight plug-in memory framework that automatically selectively preserves keyframes associated with task-relevant state changes for VLA policies. KEMO combines robot kinematics with visual filtering to detect events, encodes the selected keyframes as compact temporally ordered memory tokens, and integrates them with current visual features through cross-attention and gated residual fusion for VLA training. The detected events also define higher-weight training samples near critical transitions. We evaluate KEMO on various real-world dual-arm manipulation tasks spanning 2 to 6 scored subtasks, and trajectory length ranging from 830 steps to 2846 execution steps (durations from 28 to 95 seconds). Compared with the memory-free baseline (e.g., $π_{0.5}$), KEMO improves aggregate Task Success Rate by 23.6\% and Stage Completion Rate by 34.1\%. Ablations show that event-driven keyframe selection outperforms uniform sampling and recent-frame retention, while the proposed gated fusion and keyframe-aligned loss weighting provide complementary gains.

Problem

• long-horizon robot manipulation
• task progress ambiguity
• memory-augmented policies
• keyframe selection
• VLA policies

Innovation

• event-driven memory
• keyframe selection
• VLA policies
• gated residual fusion
• long-horizon manipulation