GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos

2026-09-15

AI Summary

本文提出GeoLAM框架，从无标签的人类视频中学习几何基础的潜在动作，通过结合未来帧重建和4D几何教师的运动监督来保留有用的运动信息。

Abstract

Human videos provide rich manipulation experience, but extracting action representations that preserve useful motion remains challenging. Visual reconstruction alone can entangle manipulation-related motion with appearance changes and camera movement. We present GeoLAM, a framework for learning geometry-grounded latent actions from action-free human videos. GeoLAM combines future-frame reconstruction through a frozen geometric feature hierarchy with motion supervision from a training-only 4D geometry teacher. The geometric representation provides a structural prior, while the teacher's predictions yield spatially pooled targets capturing 3D displacement, residual image-plane motion, and surface-orientation changes. Visibility and confidence weighting reduces the contribution of unreliable estimates, encouraging continuous latent actions to retain geometric motion without explicit hand-pose or hand-trajectory annotations. After video pretraining without action labels, the learned representation provides transition targets for a world-action model trained on action-labeled robot demonstrations. The model jointly denoises latent actions and executable action chunks, with future-video prediction used only as an auxiliary training task. Deployment therefore requires neither the geometry teacher nor future-video generation. Evaluations on a latent-action benchmark and robotic manipulation tasks demonstrate the strong performance of GeoLAM.

Problem

• unlabeled human videos
• action representations
• visual reconstruction
• manipulation-related motion

Innovation

• geometry-grounded latent actions
• frozen geometric feature hierarchy
• 4D geometry teacher
• spatially pooled targets
• visibility and confidence weighting