VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation

2025-10-17

AI Summary

Most existing robotic imitation learning approaches rely heavily on point-cloud inputs, with limited exploration of purely vision-based methods. This paper introduces VO-DP: the first single-view image-driven diffusion policy for robotic manipulation that operates without point clouds. VO-DP adaptively fuses semantic features from DINOv2 encodings and geometric features extracted via alternating attention, integrating cross-attention mechanisms with CNN-based spatial compression to construct a robust visual policy network. In simulation, VO-DP achieves a success rate of 64.6%, matching the point-cloud baseline DP3 (64.0%); in real-world experiments, it attains 87.9%, significantly outperforming DP3 (67.5%) and DP (11.2%). To our knowledge, VO-DP is the first method to enable end-to-end co-modeling of semantic and geometric features in pure-vision imitation learning, demonstrating strong environmental robustness. We open-source a scalable robot manipulation library supporting distributed training.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In the context of imitation learning, visuomotor-based diffusion policy learning is one of the main directions in robotic manipulation. Most of these approaches rely on point clouds as observation inputs and construct scene representations through point clouds feature learning, which enables them to achieve remarkable accuracy. However, the existing literature lacks an in-depth exploration of vision-only solutions that have significant potential. In this paper, we propose a Vision-Only and single-view Diffusion Policy learning method (VO-DP) that leverages pretrained visual foundation models to achieve effective fusion of semantic and geometric features. We utilize intermediate features from VGGT incorporating semantic features from DINOv2 and geometric features from Alternating Attention blocks. Features are fused via cross-attention and spatially compressed with a CNN to form the input to the policy head. Extensive experiments demonstrate that VO-DP not only outperforms the vision-only baseline DP significantly but also exhibits distinct performance trends against the point cloud-based method DP3: in simulation tasks, VO-DP achieves an average success rate of 64.6% on par with DP3 64.0% and far higher than DP 34.8%, while in real-world tasks, it reaches 87.9%, outperforming both DP3 67.5% and DP 11.2% by a notable margin. Further robustness evaluations confirm that VO-DP remains highly stable under varying conditions including color, size, background, and lighting. Lastly, we open-source a training library for robotic manipulation. Built on Accelerate, this library supports multi-machine and multi-GPU parallel training, as well as mixed precision training. It is compatible with visuomotor policies such as DP, DP3 and VO-DP, and also supports the RoboTwin simulator.

Problem

• Develops vision-only robotic manipulation without point clouds
• Fuses semantic and geometric features using pretrained models
• Achieves high success rates in simulation and real-world tasks

Innovation

• Vision-only diffusion policy with semantic-geometric fusion
• Cross-attention fusion of VGGT, DINOv2, and Alternating Attention features
• CNN spatial compression for policy input generation