Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

2026-06-18

AI Summary

This work addresses the limited out-of-distribution generalization of vision-language-action (VLA) policies to novel objects and the high cost of collecting new multi-view teleoperated data. The authors propose a failure-driven data augmentation framework that requires no additional data collection: by replacing the manipulated object in successful trajectories with novel objects in 3D space while preserving the original six-degree-of-freedom (6D) pose trajectory, the method generates physically plausible and multi-view consistent training samples. This approach uniquely integrates explicit 3D meshes, temporally consistent 6D pose estimation, and multi-view rendering to achieve geometrically and physically coherent object replacement under strong occlusions and egocentric viewpoints. Experiments demonstrate that fine-tuning VLA policies with the augmented data improves success rates on novel objects by 16.5% over the current best baseline, without degrading performance on the original data distribution.

Abstract

Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. The standard remedy is to collect multi-view teleoperation data for every failure case, but this scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven data augmentation framework that turns a policy's own successful episodes into targeted demonstrations for its failure modes, without any new data collection. Our key insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. By swapping only the manipulated object while preserving this trajectory, we obtain new and physically grounded demonstrations. However, naive 2D video editing breaks multi-view consistency and physical plausibility, particularly under heavy occlusion and egocentric viewpoints. Our method instead operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory, ensuring geometrically consistent renderings across all camera views. Fine-tuning a VLA on data augmented by our method improves success rates by 16.5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance. These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization.

Problem

• out-of-distribution generalization
• robot manipulation
• data augmentation
• multi-view consistency
• physical plausibility

Innovation

• data augmentation
• 6D pose
• multi-view consistency
• physically plausible
• vision-language-action