A Study on Enhancing the Generalization Ability of Visuomotor Policies via Data Augmentation

2025-11-13

AI Summary

Visual motor policies often exhibit poor generalization in real-world scenarios—such as object randomization—due to domain gaps between simulation and reality. Method: This paper proposes a large-scale, multi-factor scene randomization data augmentation framework for zero-shot simulation-to-reality (Sim2Real) transfer. It jointly randomizes six environmental dimensions—robot arm configuration, gripper morphology, camera pose, illumination, texture, and table height—while automatically generating diverse manipulation trajectories from minimal human demonstrations. Results: Evaluated across six robotic manipulation tasks on low-cost hardware, the method significantly mitigates visual discrepancies between simulation and reality. All randomized factors improve generalization, with trajectory diversity proving especially critical for bridging visual domain gaps. Crucially, it enables zero-shot cross-scenario transfer without any real-world fine-tuning, achieving robust policy deployment under unseen object arrangements and environmental variations.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The generalization ability of visuomotor policy is crucial, as a good policy should be deployable across diverse scenarios. Some methods can collect large amounts of trajectory augmentation data to train more generalizable imitation learning policies, aimed at handling the random placement of objects on the scene's horizontal plane. However, the data generated by these methods still lack diversity, which limits the generalization ability of the trained policy. To address this, we investigate the performance of policies trained by existing methods across different scene layout factors via automate the data generation for those factors that significantly impact generalization. We have created a more extensively randomized dataset that can be efficiently and automatically generated with only a small amount of human demonstration. The dataset covers five types of manipulators and two types of grippers, incorporating extensive randomization factors such as camera pose, lighting conditions, tabletop texture, and table height across six manipulation tasks. We found that all of these factors influence the generalization ability of the policy. Applying any form of randomization enhances policy generalization, with diverse trajectories particularly effective in bridging visual gap. Notably, we investigated on low-cost manipulator the effect of the scene randomization proposed in this work on enhancing the generalization capability of visuomotor policies for zero-shot sim-to-real transfer.

Problem

• Enhancing visuomotor policy generalization through automated data augmentation
• Addressing limited diversity in existing trajectory augmentation methods
• Investigating scene layout factors for zero-shot sim-to-real transfer

Innovation

• Automated data generation for key generalization factors
• Extensively randomized dataset from minimal human demonstration
• Enhanced policy generalization via diverse trajectory augmentation