KAM-WM: Kinematic Affordance Maps from Latent World Models for Robot Manipulation

2026-07-06

AI Summary

This work addresses the lack of task-relevant, directional visual priors in existing methods for learning robotic manipulation from few demonstrations. The authors propose a novel approach that leverages a frozen latent video world model to extract motion priors through a single query to a Flow Matching image-to-video backbone, parsing its one-step latent velocity into a Kinematic Affordance Map (KAM) that encodes task-conditioned interaction regions and coarse motion structure. This directional prior—extracted without forward simulation or fine-tuning—is combined with a lightweight Perceiver encoder and a diffusion policy for efficient control. Evaluated on LIBERO, the method achieves an average success rate of 90.6%, and on RoboTwin2.0 reaches 65.7% and 22.4% in Easy and Hard settings, respectively, substantially outperforming baselines relying solely on static mask priors.

Abstract

Learning manipulation from few demonstrations requires visual priors that capture not only where to interact, but also how the interaction should begin; static priors such as segmentation masks encode only the former. We present KAM-WM, a framework that extracts a coarse directional interaction cue from a frozen latent video world model without rollout or world-model fine-tuning. KAM-WM queries a Flow Matching image-to-video backbone once and interprets its single-step latent velocity as a Kinematic Affordance Map (KAM), which provides task-conditioned interaction regions and coarse motion structure. A lightweight Perceiver compresses KAM into tokens that condition a diffusion policy together with RGB observations and proprioception. Across LIBERO and RoboTwin2.0, KAM-WM reaches 90.6% average success on LIBERO and achieves 65.7% and 22.4% success rates in the Easy and Hard settings on RoboTwin2.0, respectively. Controlled comparisons against a zero-order mask prior suggest that part of the gains comes from directional information beyond spatial localization alone. These results indicate that, in the evaluated settings, a frozen video model can provide a useful first-order visual prior for control without the test-time cost of future rollout.

Problem

• robot manipulation
• visual priors
• kinematic affordance
• few-shot learning
• directional interaction

Innovation

• Kinematic Affordance Map
• Latent World Model
• Flow Matching
• Diffusion Policy
• Visual Prior