Flow-aware Optimal Navigation in Unsteady Flows through Reinforcement Learning

2026-07-15

AI Summary

This work addresses the challenge of autonomous navigation in unsteady, time-varying fluid environments characterized by partial observability and inherent unpredictability. The authors propose a navigation strategy based on Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning that relies solely on local flow velocity, vorticity, and short-term memory to achieve goal-directed navigation within a parameterized chaotic double-gyre flow field. Systematic evaluation of various bio-inspired sensing modalities reveals that explicit access to global flow parameters degrades performance, whereas agents integrating local velocity and vorticity perception achieve optimal results: velocity sensing enhances energy efficiency, while vorticity sensing improves mapping of flow structures and accuracy in target approach. These findings demonstrate that robust navigation in complex flows is better supported by implicit, local sensory cues rather than global flow information.

Abstract

Autonomous robotic navigation in nonstationary time-varying fluid flows remains a fundamental challenge due to partial observability and the unpredictability of realistic environments. While classical optimal control frameworks employed in robotics require unrealistic a-priori global flow knowledge, biological systems are able to navigate successfully by exploiting localized sensory cues. In this work we present a reinforcement learning approach using the TD3 algorithm to train autonomous agents to reach arbitrary targets within a parametric, chaotic double-gyre flow. To investigate optimal sensory mechanisms, we evaluate five bio-inspired observation strategies based on relative position, local velocity or local vorticity measures, and short-term memory variants. Additionally, we analyze the impact of providing agents with explicit global flow parameters. Numerical results demonstrate that an agent that is able to sense and remember a set number of flow velocity measures achieves the highest performance. The experiments reveal a trade-off in sensor utility: velocity-aware agents optimize energy efficiency, whereas vorticity sensors provide superior structural mapping and achieve better target proximity. Incorporating explicit global flow parameters is shown to decrease navigation performance. This behavior suggests that reinforcement learning-based autonomous systems develop more robust and general policies when restricted to implicit flow representations. The presented results offer insights for improving the transition of bio-inspired robotic navigation from simulation to real-world environments.

Problem

• unsteady flows
• autonomous navigation
• partial observability
• bio-inspired sensing
• optimal control

Innovation

• reinforcement learning
• bio-inspired sensing
• unsteady flows
• autonomous navigation
• implicit flow representation