Disengagement Analysis and Field Tests of a Prototypical Open-Source Level 4 Autonomous Driving System

2026-03-23

AI Summary

This study addresses the lack of systematic evaluation of disengagement behaviors in open-source Level 4 autonomous driving systems during long-distance real-world operation. Building a vehicle platform based on Autoware, the authors conducted 26 test runs covering 236 kilometers in mixed-traffic environments and propose the first five-tier criticality framework for classifying and analyzing 30 disengagement events. Introducing the spatial disengagement rate (0.127 incidents per kilometer) and a human intervention logging methodology, the study reveals robustness deficiencies in scenarios involving static obstacles and traffic signals: 40% of disengagements stemmed from perception failures such as target tracking loss, while 26.7% resulted from planning deadlocks. Additionally, frequent unnecessary interventions by safety drivers were attributed to insufficient trust in the system. This work establishes a novel paradigm for safety assessment of open-source autonomous driving systems.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Multiagent Systems: Adversarial Agents
• Intelligent Robots: Localization, Mapping, and Navigation

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Proprietary Autonomous Driving Systems are typically evaluated through disengagements, unplanned manual interventions to alter vehicle behavior, as annually reported by the California Department of Motor Vehicles. However, the real-world capabilities of prototypical open-source Level 4 vehicles over substantial distances remain largely unexplored. This study evaluates a research vehicle running an Autoware-based software stack across 236 km of mixed traffic. By classifying 30 disengagements across 26 rides with a novel five-level criticality framework, we observed a spatial disengagement rate of 0.127 1/km. Interventions predominantly occurred at lower speeds near static objects and traffic lights. Perception and Planning failures accounted for 40% and 26.7% of disengagements, respectively, largely due to object-tracking losses and operational deadlocks caused by parked vehicles. Frequent, unnecessary interventions highlighted a lack of trust on the part of the safety driver. These results show that while open-source software enables extensive operations, disengagement analysis is vital for uncovering robustness issues missed by standard metrics.

Problem

• disengagement
• autonomous driving
• open-source
• Level 4
• field testing

Innovation

• disengagement analysis
• open-source autonomous driving
• five-level criticality framework
• Autoware
• field testing