Multi-AUV Ad-hoc network-based Target Tracking: A Value Gradient Guidance Multi-Agent Diffusion Reinforcement Learning Approach

2026-08-12

AI Summary

This work addresses the challenges of high-dimensional modeling and policy sensitivity to noise encountered by multiple autonomous underwater vehicles (AUVs) when cooperatively tracking maneuvering targets under acoustically constrained communication, dynamically changing network topologies, and oceanic disturbances. To tackle these issues, the authors propose a diffusion-based three-layer closed-loop control architecture, termed MDCA, along with a novel multi-agent diffusion reinforcement learning algorithm, VGG-MADiffRL, which incorporates value-gradient guidance into the denoising process of diffusion policies. By integrating dual value networks and soft target updates, the algorithm enhances action quality and training stability. Experimental results demonstrate that the proposed approach achieves faster convergence, higher tracking accuracy, and smoother training dynamics in dynamic underwater environments, significantly improving the stability and practical applicability of multi-AUV cooperative target tracking.

Abstract

Multi-AUV ad-hoc network-based target tracking requires networked autonomous underwater vehicles (AUVs) to cooperatively track maneuvering targets under constrained acoustic communication, dynamic topology, and uncertain ocean disturbances. Although multi-agent reinforcement learning (MARL) enables decentralized coordination through centralized training, existing methods suffer from high-dimensional joint state-action modeling, noise-sensitive policy generation, leading to unstable training and degraded tracking. To address these issues, we propose VGG-MADiffRL, a value-gradient-guided multi-agent diffusion RL algorithm, and MDCA, a diffusion?based hierarchical control architecture. Leveraging underwater mission characteristics, we model sonar detection mechanisms and ocean current disturbances, formulating cooperative tracking for multi-AUV ad-hoc networks as an MDP. The proposed MDCA constitutes a three-tier closed-loop control framework: a global intelligent control layer, a local online training layer, and a physical action execution layer. This structure enables synergistic optimization across task allocation, local decision processes, and execution feedback. Within MDCA, the local online training layer is the policy learning framework; VGG-MADiffRL builds on diffusion policies and incorporates value gradients to guide action generation in the reverse denoising process, steering the generated actions towards higher expected returns. It employs twin value networks with joint optimization and soft target updates to mitigate overestimation and training oscillations, promoting more stable convergence. Experimental results show that VGG-MADiffRL consistently achieves faster convergence, higher tracking accuracy, and smoother training dynamics in cooperative tracking scenarios, validating its effectiveness and practical engineering value in dynamic underwater settings.

Problem

• Multi-AUV
• Ad-hoc network
• Target tracking
• Underwater communication
• Ocean disturbances

Innovation

• diffusion reinforcement learning
• value gradient guidance
• multi-AUV ad-hoc network
• hierarchical control architecture
• cooperative target tracking