Flying Hand: End-Effector-Centric Framework for Versatile Aerial Manipulation Teleoperation and Policy Learning

2025-04-14

AI Summary

Existing aerial manipulation frameworks suffer from tight coupling between tasks and platforms, hindering algorithm generalization and standardization. This paper introduces the first end-to-end actuator-centric aerial manipulation framework that decouples platform-agnostic high-level decision-making from task-agnostic low-level control. Our key contributions are: (1) a full-body model predictive controller (MPC) formulated at the actuator level, enabling high-fidelity end-effector tracking while achieving hardware–algorithm decoupling; and (2) a learning-based policy architecture grounded in imitation learning, supporting cross-task and cross-platform teleoperation as well as autonomous skill acquisition. Evaluated on a fully-actuated hexarotor equipped with a 4-DOF manipulator, our framework reduces end-effector positioning error by 42% and successfully executes diverse manipulation tasks—including handwriting, peg-in-hole insertion, pick-and-place, and lightbulb replacement—demonstrating a significant step toward general-purpose robotic manipulation in aerial systems.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Aerial manipulation has recently attracted increasing interest from both industry and academia. Previous approaches have demonstrated success in various specific tasks. However, their hardware design and control frameworks are often tightly coupled with task specifications, limiting the development of cross-task and cross-platform algorithms. Inspired by the success of robot learning in tabletop manipulation, we propose a unified aerial manipulation framework with an end-effector-centric interface that decouples high-level platform-agnostic decision-making from task-agnostic low-level control. Our framework consists of a fully-actuated hexarotor with a 4-DoF robotic arm, an end-effector-centric whole-body model predictive controller, and a high-level policy. The high-precision end-effector controller enables efficient and intuitive aerial teleoperation for versatile tasks and facilitates the development of imitation learning policies. Real-world experiments show that the proposed framework significantly improves end-effector tracking accuracy, and can handle multiple aerial teleoperation and imitation learning tasks, including writing, peg-in-hole, pick and place, changing light bulbs, etc. We believe the proposed framework provides one way to standardize and unify aerial manipulation into the general manipulation community and to advance the field. Project website: https://lecar-lab.github.io/flying_hand/.

Problem

• Decouples high-level decision-making from low-level control in aerial manipulation
• Enables versatile aerial teleoperation and imitation learning for diverse tasks
• Improves end-effector tracking accuracy across multiple real-world applications

Innovation

• End-effector-centric whole-body model predictive controller
• Decoupled high-level platform-agnostic decision-making
• Fully-actuated hexarotor with 4-DoF robotic arm