Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty

2026-06-15

AI Summary

This study addresses the challenge of whole-body direct control for aerial manipulators under rapid motion, varying payloads, and dynamic uncertainties. The authors propose a hierarchical control framework in which an outer loop employs reinforcement learning to map desired six-degree-of-freedom end-effector trajectories into coordinated whole-body commands, while an inner loop incorporates a model-free dynamics estimator to compensate in real time for abrupt inertial changes and system uncertainties. This approach uniquely integrates reinforcement learning with a model-free inner-loop estimator, eliminating reliance on an accurate coupled dynamics model and substantially enhancing system robustness. Hardware-in-the-loop experiments on a physical quadrotor platform demonstrate significantly reduced end-effector tracking errors and markedly improved task success rates compared to baseline methods such as RL+PID.

Abstract

Aerial manipulators enable physical interaction in hard-to-reach environments; however, the combined problem of direct whole-body aerial manipulation under rapid arm motion, payload changes, and related unknown dynamic uncertainty remains a largely unsolved problem. We present a hierarchical control framework that combines Reinforcement Learning (RL) with an inner-loop dynamics estimator to address this problem. The RL outer loop maps desired 6-degrees-of-freedom (DOF) end-effector targets to coordinated whole-body commands, enabling direct task-driven control without relying on a fully accurate coupled dynamic model in the policy layer. An inner loop then tracks these commands while compensating for transient inertial shifts and uncertainty during execution via a dynamics estimator scheme without requiring system model knowledge. We validate the proposed approach on a custom quadrotor equipped with a 3-DoF manipulator through hardware experiments under varying payload conditions. Compared with RL+PID and RL+INDI+PID baselines, the proposed method reduces end-effector tracking error and improves task success rate across the tested hardware conditions. These results show that combining learned whole-body coordination with estimator-based low-level compensation improves the precision and robustness of aerial manipulation under changing operating conditions.

Problem

• aerial manipulation
• dynamic uncertainty
• whole-body control
• payload variation
• reinforcement learning

Innovation

• Reinforcement Learning
• Dynamics Estimator
• Aerial Manipulation
• Hierarchical Control
• Model-free Compensation