Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning

2025-11-25

AI Summary

Aerodynamic parameter uncertainty in flight testing poses significant maneuver safety risks, yet existing abort criteria lack theoretical guarantees and struggle to handle dynamic uncertainties. Method: We propose a data-driven real-time safety alerting framework comprising three stages: trajectory prediction, nearest-neighbor safety classification, and conformal prediction–based calibration—enabling reliable quantification of short-term safety risk. Contribution/Results: To our knowledge, this is the first work to integrate conformal prediction into flight safety classification calibration, providing rigorous coverage probability guarantees under user-specified confidence levels and enabling cross-configuration generalization. Experiments on uncertain flight dynamic models demonstrate that the system accurately identifies critical hazardous scenarios, achieving significantly higher risk anticipation accuracy than baseline methods while strictly satisfying theoretical coverage requirements.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We develop a data-driven approach for runtime safety monitoring in flight testing, where pilots perform maneuvers on aircraft with uncertain parameters. Because safety violations can arise unexpectedly as a result of these uncertainties, pilots need clear, preemptive criteria to abort the maneuver in advance of safety violation. To solve this problem, we use offline stochastic trajectory simulation to learn a calibrated statistical model of the short-term safety risk facing pilots. We use flight testing as a motivating example for data-driven learning/monitoring of safety due to its inherent safety risk, uncertainty, and human-interaction. However, our approach consists of three broadly-applicable components: a model to predict future state from recent observations, a nearest neighbor model to classify the safety of the predicted state, and classifier calibration via conformal prediction. We evaluate our method on a flight dynamics model with uncertain parameters, demonstrating its ability to reliably identify unsafe scenarios, match theoretical guarantees, and outperform baseline approaches in preemptive classification of risk.

Problem

• Develops runtime safety monitoring for flight testing with uncertain parameters
• Provides preemptive criteria to abort maneuvers before safety violations occur
• Uses data-driven approach to classify safety risks in uncertain flight scenarios

Innovation

• Offline stochastic simulation for safety risk modeling
• Nearest neighbor classification for predicted state safety
• Classifier calibration via conformal prediction method