One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies

2026-06-17

AI Summary

This work addresses the failure of visuomotor policies under observation distribution shifts caused by changes in initial states or encounters with previously unseen obstacles. To mitigate this issue, the authors propose a data augmentation framework that leverages a single demonstration captured by a fisheye camera. The approach innovatively integrates wide-field Gaussian splatting for 3D reconstruction, trajectory optimization, and a novel view synthesis technique to generate realistic multi-view visual sequences containing novel objects along with their corresponding executable action trajectories. Experimental results demonstrate that the proposed framework substantially improves policy success rates across diverse manipulation tasks in both simulation and real-world settings, with particularly strong performance in generalization scenarios requiring obstacle avoidance.

Abstract

Visuomotor policies for manipulation have demonstrated remarkable potential in modeling complex robotic behaviors, yet minor alterations in the robot's initial configuration and unseen obstacles easily lead to out-of-distribution observations. Without extensive data collection effort, these result in catastrophic execution failures. In this work, we introduce an effective data augmentation framework that generates visually realistic fisheye image sequences and corresponding physically feasible action trajectories from real-world eye-in-hand demonstrations, captured with a portable parallel gripper with a single fisheye camera. We introduce a novel Gaussian Splatting formulation, adapted to wide FoV fisheye cameras, to reconstruct and edit the 3D scene with unseen objects. We utilize trajectory optimization to generate smooth, collision-free, view-rendering-friendly action trajectories and render visual observations from corresponding novel views. Comprehensive experiments in simulation and the real world show that our augmentation framework improves the success rate for various manipulation tasks in both the same scene and the augmented scene with obstacles requiring collision avoidance.

Problem

• visuomotor policies
• out-of-distribution
• data augmentation
• robotic manipulation
• fisheye camera

Innovation

• Action-View Augmentation
• Gaussian Splatting
• Fisheye Camera
• Visuomotor Policy
• Trajectory Optimization