Is FISHER All You Need in The Multi-AUV Underwater Target Tracking Task?

2024-12-05

arXiv.org

AI Summary

In multi-AUV cooperative underwater target tracking, reinforcement learning (RL) suffers from excessive environmental interaction requirements, challenging reward engineering, and poor generalization of conventional controllers. To address these issues, this paper proposes FISHER, a two-stage imitation learning framework. Its key contributions are: (1) the first multi-agent adversarial imitation learning approach grounded in Nash equilibrium; (2) an independent generalized decision transformer enabling knowledge transfer from classical controllers to RL policies; and (3) a sim-to-sim demonstration generation pipeline coupled with a multi-agent discriminator–actor–critic architecture. Extensive simulations across diverse scenarios demonstrate that FISHER significantly improves tracking accuracy and coordination efficiency, while exhibiting strong robustness, cross-task generalization capability, and runtime stability.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

It is significant to employ multiple autonomous underwater vehicles (AUVs) to execute the underwater target tracking task collaboratively. However, it's pretty challenging to meet various prerequisites utilizing traditional control methods. Therefore, we propose an effective two-stage learning from demonstrations training framework, FISHER, to highlight the adaptability of reinforcement learning (RL) methods in the multi-AUV underwater target tracking task, while addressing its limitations such as extensive requirements for environmental interactions and the challenges in designing reward functions. The first stage utilizes imitation learning (IL) to realize policy improvement and generate offline datasets. To be specific, we introduce multi-agent discriminator-actor-critic based on improvements of the generative adversarial IL algorithm and multi-agent IL optimization objective derived from the Nash equilibrium condition. Then in the second stage, we develop multi-agent independent generalized decision transformer, which analyzes the latent representation to match the future states of high-quality samples rather than reward function, attaining further enhanced policies capable of handling various scenarios. Besides, we propose a simulation to simulation demonstration generation procedure to facilitate the generation of expert demonstrations in underwater environments, which capitalizes on traditional control methods and can easily accomplish the domain transfer to obtain demonstrations. Extensive simulation experiments from multiple scenarios showcase that FISHER possesses strong stability, multi-task performance and capability of generalization.

Problem

• Proposes FISHER framework for multi-AUV underwater target tracking
• Addresses reinforcement learning limitations in environmental interaction requirements
• Solves challenges in designing reward functions for collaborative AUV control

Innovation

• Two-stage learning from demonstrations training framework
• Multi-agent discriminator-actor-critic with Nash equilibrium optimization
• Generalized decision transformer analyzing latent representations of states