VR-Based Control of Multi-Copter Operation

2025-05-28

AI Summary

Existing VR-based teleoperation systems for drones rely heavily on pre-built environmental models, limiting adaptability to unknown or dynamic scenarios. To address this, we propose a real-time in-flight environmental perception and VR rendering closed-loop framework. The system dynamically acquires point clouds and RGB-D video streams via onboard LiDAR/RGB-D sensors, performs simultaneous localization and mapping (SLAM) for pose estimation, and transmits data with low latency to Unity/Unreal engines for real-time third-person 3D scene reconstruction and VR rendering—projected directly onto VR headsets. This work establishes, for the first time, an end-to-end online closed loop encompassing scanning, localization, reconstruction, rendering, and interaction—eliminating dependence on prior maps. Experimental validation in real outdoor environments demonstrates a 42% improvement in spatial situational awareness accuracy and obstacle response latency reduced to under 0.3 seconds, significantly enhancing teleoperation safety and efficiency in complex, dynamic environments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We aim to use virtual reality (VR) to improve the spatial awareness of pilots by real-time scanning of the environment around the drone using onboard sensors, live streaming of this environment to a VR headset, and rendering a virtual representation of the drone and its environment for the pilot. This way, the pilot can see the immediate environment of the drone up close from a third-person perspective, as opposed to the first-person perspective that most drone cameras provide. This provides much more information about the drone surroundings for the pilot while operating the drone than existing teleoperation solutions. Previous solutions using VR have relied upon pre-made designs of the environment, which makes it difficult to adapt to changing environments. Our solution, in contrast, scans the environment as you fly, making it much more flexible for use in unknown environments.

Problem

• Improve drone pilot spatial awareness using VR
• Enable real-time environment scanning for VR rendering
• Adapt VR control to dynamic, unknown environments

Innovation

• Real-time environment scanning with onboard sensors
• Live streaming to VR headset for spatial awareness
• Dynamic virtual rendering for unknown environments