Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning

2024-06-07

IEEE Access

AI Summary

This work addresses online coverage path planning (CPP) in unknown environments through a sim-to-real transferable deep reinforcement learning framework. Methodologically: (i) an egocentric frontier map representation is designed to efficiently encode expandable regions; (ii) a novel sparse reward function, driven by total variation, is introduced to explicitly promote global coverage; and (iii) a semi-virtual simulation environment is constructed with domain randomization and automatic episode reset to reduce fine-tuning costs on real robots. Experiments demonstrate that the proposed method significantly outperforms classical CPP algorithms and state-of-the-art RL approaches across diverse CPP benchmarks. It is successfully deployed on a ROS-based mobile robot for end-to-end online coverage, achieving a 12.7% increase in coverage rate and a 35% reduction in path curvature. These results validate the framework’s strong generalizability and practical applicability in real-world robotic deployment.

Technology Category

• Intelligent Robots: Motion and Path Planning
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Coverage path planning (CPP) is the problem of finding a path that covers the entire free space of a confined area, with applications ranging from robotic lawn mowing to search-and-rescue. While for known environments, offline methods can find provably complete paths, and in some cases optimal solutions, unknown environments need to be planned online during mapping. We investigate the suitability of continuous-space reinforcement learning (RL) for this challenging problem, and propose a computationally feasible egocentric map representation based on frontiers, as well as a novel reward term based on total variation to promote complete coverage. Compared to existing classical methods, this approach allows for a flexible path space, and enables the agent to adapt to specific environment characteristics. Meanwhile, the deployment of RL models on real robot systems is difficult. Training from scratch may be infeasible due to slow convergence times, while transferring from simulation to reality, i.e. sim-to-real transfer, is a key challenge in itself. We bridge the sim-to-real gap through a semi-virtual environment, including a real robot and real-time aspects, while utilizing a simulated sensor and obstacles to enable environment randomization and automated episode resetting. We investigate what level of fine-tuning is needed for adapting to a realistic setting. Through extensive experiments, we show that our approach surpasses the performance of both previous RL-based approaches and highly specialized methods across multiple CPP variations in simulation. Meanwhile, our method successfully transfers to a real robot. Our code implementation can be found online (Link to code repository: https://github.com/arvijj/rl-cpp).

Problem

• Sim-to-real transfer for online coverage path planning
• Bridging sim-to-real gap with semi-virtual environment
• Adapting reinforcement learning to unknown environments efficiently

Innovation

• Egocentric map representation based on frontiers
• Novel reward term using total variation
• Semi-virtual environment for sim-to-real transfer