Rich State Observations Empower Reinforcement Learning to Surpass PID: A Drone Ball Balancing Study

2025-09-25

AI Summary

This paper addresses the control problem of balancing a spherical object on a movable beam driven by UAV-mounted cables. To overcome the performance limitations of conventional PID controllers—constrained by low-dimensional state perception—we propose a hierarchical reinforcement learning (HRL) framework: a high-level policy learns balance actions from rich, high-dimensional state observations (including beam/sphere pose and velocity, cable tension, etc.), while a low-level PID controller executes UAV trajectory tracking. Experiments in simulation demonstrate that the HRL approach significantly outperforms finely tuned PID baselines. Crucially, the key contribution lies in identifying the root source of RL’s superiority—not nonlinear function approximation or hyperparameter optimization—but rather its ability to exploit comprehensive state information for superior closed-loop feedback. This finding empirically validates that enhanced perception is decisive for improving performance in learning-based control systems.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

This paper addresses a drone ball-balancing task, in which a drone stabilizes a ball atop a movable beam through cable-based interaction. We propose a hierarchical control framework that decouples high-level balancing policy from low-level drone control, and train a reinforcement learning (RL) policy to handle the high-level decision-making. Simulation results show that the RL policy achieves superior performance compared to carefully tuned PID controllers within the same hierarchical structure. Through systematic comparative analysis, we demonstrate that RL's advantage stems not from improved parameter tuning or inherent nonlinear mapping capabilities, but from its ability to effectively utilize richer state observations. These findings underscore the critical role of comprehensive state representation in learning-based systems and suggest that enhanced sensing could be instrumental in improving controller performance.

Problem

• Developing a hierarchical control framework for drone ball-balancing tasks
• Comparing reinforcement learning performance against tuned PID controllers
• Investigating how richer state observations improve controller performance

Innovation

• Hierarchical framework decouples high-level balancing policy
• Reinforcement learning handles high-level decision-making
• Utilizes richer state observations for superior performance