An expert ensemble for detecting anomalous scenes, interactions, and behaviors in autonomous driving

2024-11-21

The international journal of robotics research

AI Summary

Unsupervised detection of out-of-distribution (OOD) scenarios, interactive anomalies, and individual behavioral anomalies in autonomous driving onboard video remains challenging due to the lack of labeled anomaly data and the complexity of spatiotemporal dynamics. Method: This paper proposes a three-level fine-grained anomaly modeling framework comprising dedicated expert modules for scene, interaction, and behavior. It introduces Xen—a novel state-estimation-based multi-expert ensemble mechanism—employing Kalman filtering for dynamic fusion. The method jointly models frame-level appearance, relative motion, and trajectory prediction to enable unsupervised, interpretable anomaly typing. Contribution/Results: Evaluated on a large-scale real-world road anomaly dataset, our approach significantly outperforms existing methods. We further establish a novel evaluation protocol grounded in safety-critical metrics, enhancing both system safety assurance and model trustworthiness.

Technology Category

• Intelligent Robots: State Estimation
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As automated vehicles enter public roads, safety in a near-infinite number of driving scenarios becomes one of the major concerns for the widespread adoption of fully autonomous driving. The ability to detect anomalous situations outside of the operational design domain is a key component in self-driving cars, enabling us to mitigate the impact of abnormal ego behaviors and to realize trustworthy driving systems. On-road anomaly detection in egocentric videos remains a challenging problem due to the difficulties introduced by complex and interactive scenarios. We conduct a holistic analysis of common on-road anomaly patterns, from which we propose three unsupervised anomaly detection experts: a scene expert that focuses on frame-level appearances to detect abnormal scenes and unexpected scene motions; an interaction expert that models normal relative motions between two road participants and raises alarms whenever anomalous interactions emerge; and a behavior expert which monitors abnormal behaviors of individual objects by future trajectory prediction. To combine the strengths of all the modules, we propose an expert ensemble (Xen) using a Kalman filter, in which the final anomaly score is absorbed as one of the states and the observations are generated by the experts. Our experiments employ a novel evaluation protocol for realistic model performance, demonstrate superior anomaly detection performance than previous methods, and show that our framework has potential in classifying anomaly types using unsupervised learning on a large-scale on-road anomaly dataset.

Problem

• Detect anomalies in autonomous driving scenarios
• Identify abnormal interactions between road participants
• Monitor and predict abnormal object behaviors

Innovation

• Unsupervised anomaly detection experts
• Expert ensemble with Kalman filter
• Future trajectory prediction for behaviors