Gesture-Controlled Aerial Robot Formation for Human-Swarm Interaction in Safety Monitoring Applications

2024-03-22

IEEE Robotics and Automation Letters

AI Summary

To address the need for contactless human–drone interaction in aerial work safety monitoring, this paper proposes a gesture-based cooperative control method for multi-rotor drone formations. The leader drone performs real-time human detection, 3D pose estimation, and markerless hand-gesture recognition; follower drones maintain prescribed relative configurations via a distributed formation control protocol, enabling multi-perspective collaborative surveillance. Our key contribution lies in the end-to-end deployment of an integrated pipeline—comprising human detection, 3D pose estimation, and lightweight gesture recognition—directly onto resource-constrained onboard hardware, eliminating reliance on external markers or dedicated communication channels. Experimental validation with a three-drone swarm and live human subjects achieves a gesture recognition accuracy of 92.3% and an average system response latency of 376 ms, while maintaining stable formation integrity—demonstrating both efficacy and practical applicability.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Teamwork, Team formation

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

This paper presents a formation control approach for contactless gesture-based Human-Swarm Interaction (HSI) between a team of multi-rotor Unmanned Aerial Vehicles (UAVs) and a human worker. The approach is designed to monitor the safety of human workers, particularly those operating at heights. In the proposed dynamic formation scheme, one UAV acts as the formation leader, equipped with sensors for detecting human workers and recognizing gestures. The follower UAVs maintain a predetermined formation relative to the worker's position, providing additional perspectives of the monitored scene. Hand gestures enable the human worker to specify movement and action commands for the UAV team and to initiate other mission-related tasks without requiring additional communication channels or specific markers. Combined with a novel unified human detection and tracking algorithm, a human position estimation method, and a gesture detection pipeline, the proposed approach represents the first instance of an HSI system incorporating all these modules onboard real-world UAVs. Simulations and field experiments involving three UAVs and a human worker in a mock-up scenario demonstrate the effectiveness and responsiveness of the proposed approach.

Problem

• Controls UAV formation via gestures for safety monitoring
• Enables human-worker interaction without extra communication channels
• Integrates detection, tracking, and gesture modules on UAVs

Innovation

• Gesture-controlled UAV formation for safety monitoring
• Leader-follower UAVs with human detection sensors
• Unified human tracking and gesture recognition algorithm