TFP: Temporally Conditioned Memory-Fusion Policies for Visuomotor Learning

2026-07-09

AI Summary

This work addresses the limitation of existing vision-language-action (VLA) policies, which rely solely on current observations for reactive decision-making and thus struggle to distinguish visually similar states with divergent action requirements in phase-dependent tasks. To overcome this, the authors propose a lightweight memory-action fusion framework featuring a dynamic memory module based on liquid time-constant networks. This module employs an event-sensitive mechanism that maintains task-progress beliefs during stable periods and promptly updates memory at critical interaction points. Memory information is then causally injected into a flow-matching action decoder via adaptive modulation. Integrated into a 3.3B-parameter VLA backbone, the approach achieves success rates of 98.75% and 93.77% on LIBERO and LIBERO-plus benchmarks, respectively, and 75.0% on the MIKASA ShellGameTouch task. Mechanistic analysis confirms significantly enhanced memory write gains around manipulation events.

Abstract

Vision--Language--Action (VLA) policies such as $π_{0.5}$ and OpenVLA perform well on many manipulation tasks, but they are often reactive: the next action is predicted from the current observation, instruction, and proprioceptive state. This assumption breaks down in stage-dependent manipulation, where visually similar states may require different actions depending on latent task progress and previous interaction outcomes. We argue that such tasks require not only memory, but dynamics-aware belief updates: the policy should preserve task progress during stable or occluded phases and revise its belief near contact, release, or subgoal transitions. We introduce Temporally Conditioned Memory-Fusion Policies (TFP), a lightweight memory-action framework for VLA backbones. TFP maintains an episode-local task-progress belief with Liquid Time-Constant dynamics and injects the updated belief directly into the flow-matching action decoder through adaptive modulation. This lets temporally accumulated context shape the generated action chunk, rather than serving only as passive history context. With a 3.3B-parameter model, TFP improves the average success rate from \(96.9\%\) to \(98.75\%\) on LIBERO and from \(91.4\%\) to \(93.77\%\) on LIBERO-plus. On the memory-focused MIKASA ShellGameTouch diagnostic, TFP achieves success up to \(75.0\%\). Mechanistic analyses show that write-gain changes near manipulation events are about \(6\times\) larger than far non-event phases, and hidden-state interventions show that the belief causally modulates generated action chunks. These results suggest that compact, event-sensitive memory dynamics can improve VLA policies under occlusion, visual perturbation, and stage-dependent task structure.

Problem

• stage-dependent manipulation
• task progress
• memory
• belief update
• visuomotor learning

Innovation

• memory-fusion
• temporal belief update
• visuomotor learning
• flow-matching policy
• event-sensitive dynamics