Power-Budgeted Underwater Vehicle Control via Constrained Reinforcement Learning

2026-06-24

AI Summary

This work addresses the challenge of high propulsion energy consumption in underwater vehicles, which operate under strict power budgets. Conventional reinforcement learning approaches often induce control oscillations that waste energy while optimizing task accuracy. To overcome this, the study introduces average propulsion power as an explicit constraint within a constrained Markov decision process formulation. Leveraging the PPO-Lagrangian algorithm, it updates a single dual variable online, enabling direct specification of power budgets in physical units without manual hyperparameter tuning. Evaluated across three vehicle types and four tasks (12 configurations total) in the MarineGym platform, the proposed method achieves the lowest energy consumption in all cases—reducing power use by 14% to 64.9% compared to task-optimization baselines—and yields the smoothest control in 10 configurations. It incurs only minor accuracy degradation under stringent power limits, substantially outperforming energy-penalized baselines.

Abstract

Underwater vehicles operate from a fixed onboard energy budget that propulsion rapidly depletes, so a controller that completes its task while drawing less thruster power directly extends mission range and endurance. Reinforcement learning yields capable model-free controllers for station-keeping and trajectory tracking, but optimizing task accuracy alone drives the policy toward oscillatory, energy-wasting actuation. The established remedy subtracts an energy penalty from the reward, yet this sets the task-power trade-off through a single weight with no physical units: a target power level cannot be specified, the weight must be re-tuned for every vehicle and task, and a mismatched weight can even raise power. This paper instead formulates energy-efficient underwater control as a constrained Markov decision process in which average thruster power is subject to an explicit budget, solved with a PPO-Lagrangian algorithm. The power level is set by declaring a budget in physical units, and a single dual variable is updated online to meet it for each vehicle and task, without manual weight search. Across three vehicles and four tasks in the MarineGym simulator, the energy-constrained policy draws the least power in all twelve settings, reducing it by 14--65\% (up to 64.9\%) over a task-only baseline and below an energy-reward baseline everywhere, while remaining the smoothest in ten settings and preserving task accuracy except in one deliberately power-limited regime. Imposing energy as an explicit constraint thus offers a tuning-free route to energy-efficient underwater control that needs no per-vehicle, per-task weight search.

Problem

• underwater vehicle control
• energy efficiency
• power budget
• constrained reinforcement learning
• thruster power

Innovation

• Constrained Reinforcement Learning
• Energy-Efficient Control
• Underwater Vehicles
• PPO-Lagrangian
• Power Budget