Contrastive Action-Image Pre-training for Visuomotor Control

2026-06-15

AI Summary

Existing robotic vision encoders struggle to achieve effective pretraining due to the scarcity of large-scale, action-annotated datasets and insufficient alignment with the visuomotor signals required by downstream control tasks. This work proposes a novel paradigm that leverages 3D hand keypoints extracted from human egocentric videos as proxies for robotic end-effector actions, enabling a unified contrastive learning objective tailored for visuomotor control. By combining large-scale self-supervised pretraining with fine-tuning on limited real robot data, the approach significantly improves success rates—by over 30%—on dexterous manipulation tasks such as folding and pouring when deployed on the Dexmate Vega and Sharpa Wave robotic hands. The method consistently outperforms state-of-the-art vision encoders like DINOv2 and SigLIP in downstream control performance.

Abstract

Existing vision encoders for robotics face a fundamental bottleneck: robotic datasets lack the scale necessary for large-scale pre-training. Prior work circumvents this data scarcity by turning to internet-scale image and language data or egocentric human video. While these models show promise, neither paradigm learns from paired vision and action data, which downstream visuomotor control policies require. However, robot trajectories, the most direct source of this paired signal, are not available at pre-training scale, motivating us to extract action signals from abundant human video instead. To this end, we introduce CAIP (Contrastive Action-Image Pre-training), a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions. By extracting 3D hand keypoints, a representation that aligns naturally with downstream robot action spaces, CAIP learns a unified action-image representation through a contrastive objective. Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders including DINOv2, SigLIP, MVP, and R3M. Evaluated on a challenging real-world dexterous manipulation setup using Dexmate Vega and Sharpa Wave hands, CAIP yields performance gains of more than 30% on tasks involving folding, pouring, and fine-grained manipulation. Our results show that our method of contrastive action-centric pre-training yields a scalable path to achieving robust visual representations better suited for physical interaction.

Problem

• visuomotor control
• action-image pairing
• robotic pre-training
• data scarcity
• vision encoder

Innovation

• contrastive learning
• action-image pre-training
• egocentric video
• visuomotor control
• 3D hand keypoints