Context-Based Meta Reinforcement Learning for Robust and Adaptable Peg-in-Hole Assembly Tasks

2024-09-24

arXiv.org

AI Summary

To address pose uncertainty in hole-axis assembly under unknown environments caused by onboard sensor errors, this paper proposes a context-driven meta-reinforcement learning (meta-RL) framework. Methodologically: (1) it designs a data representation grounded in easily measurable physical features; (2) it introduces the first meta-RL architecture that replaces vision with six-axis force/torque sensing; and (3) it develops a robust online fine-tuning mechanism generalizable to hole pose parameter deviations up to ten times the nominal range. The approach integrates a PPO-MAML variant, multimodal sensing (uncalibrated camera + tactile force feedback), and a safety-constrained context encoder. Experiments demonstrate 100% assembly success rates in both simulation and real-robot settings; real-world sample efficiency improves tenfold; and the method successfully generalizes to out-of-distribution tasks.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Peg-in-hole assembly in unknown environments is a challenging task due to onboard sensor errors, which result in uncertainty and variations in task parameters such as the hole position and orientation. Meta Reinforcement Learning (Meta RL) has been proposed to mitigate this problem as it learns how to quickly adapt to new tasks with different parameters. However, previous approaches either depend on a sample-inefficient procedure or human demonstrations to perform the task in the real world. Our work modifies the data used by the Meta RL agent and uses simple features that can be easily measured in the real world even with an uncalibrated camera. We further adapt the Meta RL agent to use data from a force/torque sensor, instead of the camera, to perform the assembly, using a small amount of training data. Finally, we propose a fine-tuning method that consistently and safely adapts to out-of-distribution tasks with parameters that differ by a factor of 10 from the training tasks. Our results demonstrate that the proposed data modification significantly enhances the training and adaptation efficiency and enables the agent to achieve 100% success in tasks with different hole positions and orientations. Experiments on a real robot confirm that both camera- and force/torque sensor-equipped agents achieve 100% success in tasks with unknown hole positions, matching their simulation performance and validating the approach's robustness and applicability. Compared to the previous work with sample-inefficient adaptation, our proposed methods are 10 times more sample-efficient in the real-world tasks.

Problem

• Addresses uncertainty in peg-in-hole assembly due to sensor errors.
• Enhances Meta RL for real-world adaptability with minimal training data.
• Improves sample efficiency and success rate in unknown environments.

Innovation

• Uses simple, real-world measurable features
• Integrates force/torque sensor data for assembly
• Proposes efficient fine-tuning for out-of-distribution tasks