ImpedanceGPT: VLM-driven Impedance Control of Swarm of Mini-drones for Intelligent Navigation in Dynamic Environment

2025-03-04

AI Summary

Safe and efficient navigation of micro-UAV swarms in dynamic human-robot collaborative environments remains challenging due to heterogeneous, time-varying obstacles. Method: This paper proposes a semantic-driven adaptive impedance control framework. It innovatively integrates vision-language models (VLMs) and retrieval-augmented generation (RAG) into the impedance control loop to enable real-time semantic perception and hierarchical response to both living (e.g., pedestrians) and non-living obstacles. Semantic class information is used to dynamically modulate stiffness and damping parameters online, overcoming the limitations of conventional geometry-based collision avoidance. Results: Experiments demonstrate 80% accuracy in obstacle identification and retrieval. In static environments, flight speed adaptively decreases from 1.4 m/s to 0.7 m/s under elevated risk; in dynamic settings, speeds adjust to 1.0 m/s for rigid obstacles and 0.6 m/s when yielding to moving humans. Average minimum separation distance improves by 40%.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Human-Robot Interaction
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Swarm robotics plays a crucial role in enabling autonomous operations in dynamic and unpredictable environments. However, a major challenge remains ensuring safe and efficient navigation in environments filled with both dynamic alive (e.g., humans) and dynamic inanimate (e.g., non-living objects) obstacles. In this paper, we propose ImpedanceGPT, a novel system that combines a Vision-Language Model (VLM) with retrieval-augmented generation (RAG) to enable real-time reasoning for adaptive navigation of mini-drone swarms in complex environments. The key innovation of ImpedanceGPT lies in the integration of VLM and RAG, which provides the drones with enhanced semantic understanding of their surroundings. This enables the system to dynamically adjust impedance control parameters in response to obstacle types and environmental conditions. Our approach not only ensures safe and precise navigation but also improves coordination between drones in the swarm. Experimental evaluations demonstrate the effectiveness of the system. The VLM-RAG framework achieved an obstacle detection and retrieval accuracy of 80 % under optimal lighting. In static environments, drones navigated dynamic inanimate obstacles at 1.4 m/s but slowed to 0.7 m/s with increased separation around humans. In dynamic environments, speed adjusted to 1.0 m/s near hard obstacles, while reducing to 0.6 m/s with higher deflection to safely avoid moving humans.

Problem

• Enabling safe navigation of drone swarms in dynamic environments.
• Integrating VLM and RAG for real-time adaptive navigation.
• Improving coordination and obstacle avoidance in drone swarms.

Innovation

• VLM-RAG integration for real-time reasoning
• Dynamic impedance control for obstacle adaptation
• Enhanced semantic understanding for swarm navigation