High-Precision Formation Control for Heterogeneous Multi-Robot Systems via Hierarchical Hybrid Physics-Informed Deep Reinforcement Learning

2026-07-03

AI Summary

This work addresses the limitations of traditional model-based control—its reliance on precise dynamics and poor handling of uncertainty—and the low sample efficiency and poor convergence often observed in end-to-end deep reinforcement learning (DRL). To overcome these challenges, the paper proposes a Hierarchical Hybrid Physics-Informed Deep Reinforcement Learning framework (HHy-PIDRL). The upper layer employs Soft Actor-Critic (SAC) to generate leader navigation policies, while the lower layer integrates high-fidelity physics-based feedforward control, PD feedback, and an adaptive DRL residual controller, establishing a synergistic model-learning paradigm for formation control. A hierarchical reward function is designed to train omnidirectional follower robots. Experimental results demonstrate 100% success rates in both navigation and formation tasks, and ablation studies confirm the proposed architecture’s superior accuracy, responsiveness, and robustness.

Abstract

Existing classical control methods commonly require precise models and struggle to cope with model uncertainties and external disturbances, while end-to-end reinforcement learning (RL) approaches suffer from low sample efficiency and poor convergence. To overcome these challenges, this paper proposes a hierarchical hybrid physics-informed deep reinforcement learning (HHy-PIDRL) framework, aiming to realize high-precision, highly responsive formation control for heterogeneous multi-robot systems (HMRSs). The proposed framework contains two layers. Specifically, first, the upper layer designs an autonomous navigation policy network for Ackermann-steering leader based on the Soft Actor-Critic (SAC) deep reinforcement learning (DRL) algorithm. Second, the lower module integrates a high-fidelity physical feed-forward controller, a classical proportional-derivative (PD) controller, and an adaptive DRL residual controller to propose an effective hybrid model and DRL (HM-DRL)-based formation control policy network. Third, a unique hierarchical reward function is designed for training Omnidirectional followers, which effectively guides agents toward a refined, stable control policy. Experimental results demonstrate that, the success rate of both the upper-layer autonomous navigation policy network and the HM-DRL based formation control policy networks reach 100%. Meanwhile, ablation experiments are conducted to verify the validity and credibility of the proposed method.

Problem

• formation control
• heterogeneous multi-robot systems
• model uncertainty
• external disturbances
• sample efficiency

Innovation

• Hierarchical Hybrid Control
• Physics-Informed Deep Reinforcement Learning
• Heterogeneous Multi-Robot Systems
• Formation Control
• Residual DRL Controller