UA-1 PH2 DECISIVE Testing Handbook: Test Methods and Benchmarking Performance Results for sUAS in Dense Urban Environments

2025-01-29

AI Summary

This work addresses the lack of systematic performance evaluation standards for small unmanned aircraft systems (sUAS) in dense urban environments. Methodologically, it introduces the first comprehensive benchmarking framework tailored to high-dynamics, multi-obstacle urban scenarios, comprising nine standardized test categories that assess eight critical capabilities—including localization robustness, RF communication resilience, and visual navigation reliability. The framework integrates GNSS/INS positioning, multi-band RF monitoring, 3D semantic modeling and simulation, multi-view synchronized video analysis, and a unified data acquisition protocol. Empirical validation across 100+ flight trials quantifies performance boundaries and failure modes of eight leading military- and industrial-grade sUAS platforms under representative urban challenges: GPS-denied conditions, narrow-alley navigation, low-light operation, and strong specular reflections. The resulting benchmark enables reproducible, scalable, and empirically grounded assessment of sUAS urban adaptability.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Search and Optimization: Sampling/Simulation-based Search
• Application Domains: Mobility, Driving & Flight

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

This report outlines all test methods and reviews all results derived from performance benchmarking of small unmanned aerial systems (sUAS) in dense urban environments conducted during Phase 2 of the Development and Execution of Comprehensive and Integrated Systematic Intelligent Vehicle Evaluations (DECISIVE) project by the University of Massachusetts Lowell (HEROES Project UA-1). Using 9 of the developed test methods, over 100 tests were conducted to benchmark the performance of 8 sUAS platforms: Cleo Robotics Dronut X1P (P = prototype), FLIR Black Hornet 3 PRS, Flyability Elios 2 GOV, Lumenier Nighthawk V3, Parrot ANAFI USA GOV, Skydio X2D, Teal Golden Eagle, and Vantage Robotics Vesper.

Problem

• Benchmarking sUAS performance
• Testing in dense urban areas
• Evaluating 8 sUAS platforms

Innovation

• sUAS performance benchmarking
• dense urban environment testing
• nine test methods utilized